Reading the Table: Sim-to-Real Game State Detection for the Chef's Hat Card Game

arXiv:2611.15524 · cs.RO, cs.CV · Submitted 2026-11-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Reading the Table".

Dev: Here's the gist of the paper. Tabletop games offer a compelling scenario to study the interaction between artificial and human agents, combining social interaction with game state tracking and physical manipulation The authors developed a fully automated synthetic data collection pipeline using re-projection of known 3D coordinates to generate labels automatically and domain randomization to bridge the reality gap This approach was used to collect a dataset of 25413 samples to finetune an object detection and a segmentation model, which are combined to assign detected cards to game-relevant zones and infer the game state When evaluated on a real NICO humanoid robot, our pipeline achieves an F1-score of 0.906 for card-to-zone assignment, thus providing the perceptual foundation for a robot that can autonomously play Chef’s Hat against human players Data Generation Pipeline The synthetic data collection pipeline is fully automated in CoppeliaSim and creates a dataset of 25413 samples consisting of paired domain-randomized and non-randomized canonical images showing the Chef’s Hat playing cards on the playing field and custom 3D-printed cardholders Each image is labeled with card identities, zone assignments, and the projected 2D image coordinates for individual cards and playing field zones, from which bounding boxes and segmentation masks can be derived Each sample further includes the 6D simulation poses of all objects, camera parameters, and a random seed, allowing the reconstruction of any scene in simulation and enabling future extensions such as card pose estimation or motion planning for physical manipulation Domain Randomization Strategy Domain randomization is employed to mitigate the reality gap by altering visual and environmental parameters of the simulation to increase the resilience of a trained model to domain shifts and thus its ability to interpret the unseen real-world domain Specifically, lighting and background textures are randomized by moving four light sources up to 2 m in either direction along two axes at a fixed height of 2.5 m, oriented randomly, and having the color of both their specular and diffuse component randomized The camera pose and field of view are also randomized by moving the camera up to 10 cm from its default position in either direction on all three axes, rotating it up to 0.2 radians in either direction around all three axes, and selecting a random field of view between 90° and 135° Furthermore, the playing field and cardholders are randomly moved and rotated on the table to reflect the changing, imprecise real-world placements Game State Inference Mechanism The game state detection pipeline divides the problem into two sub-tasks: Object detection to locate and classify individual cards in view, and segmentation to identify different game-relevant zones of the observation image that the cards have to be assigned to These tasks are solved by finetuning both the YOLO26n and YOLO26n-seg models from Ultralytics for object detection and instance segmentation on our domain-randomized dataset, using the canonical data as the validation set For each card detected by the object detection model, we take the center of the predicted bounding box and check whether it lies within the segmentation mask of one of the game-relevant zones predicted by the segmentation model, thereby enabling us to assign individual cards to game zones Experimental Results and Conclusion The evaluation on a real-world test set of 100 rectified images recorded from the left fisheye camera of a real NICO robot under 5 conditions varying in lighting and card placement shows that our domain randomization approach greatly increases the zero-shot sim-to-real performance of a model finetuned exclusively on synthetic data The baseline models finetuned exclusively on canonical synthetic data are unable to fully assess the game state with an F1-score of 0.627, missing 7.02 cards on average In contrast, our domain-randomized models reduce the missing card average to 1.52 and achieve an F1-score of 0.906 This demonstrates the resilience of our model The ultimate goal is to integrate our perception pipeline with an embodied agent such as ELMiRA

4: to enable fully autonomous play of Chef’s Hat, combining game state inference with physical card manipulation and social interaction with human players Pruning redundant reasoning tokens The authors hypothesize that exploiting game logic constraints can lead to a sufficiently accurate game state representation in a real-world scenario without a perfect detection rate This is considered an application of the frame axiom: The game state can only change through player actions permitted by the game rules during their turn The misdetection rate of approximately 1.5 cards implies that a single observation may not provide a perfect representation of the game state However, human players are also unlikely to capture the complete game state with a single glance Instead, they continuously update their internal representation of the game while observing other players’ actions A robotic player can focus on the few cards played by other players at a time and incrementally update the game state How the training signal is built The dataset consists of 25413 samples consisting of paired domain-randomized and non-randomized canonical images showing the Chef’s Hat playing cards on the playing field and custom 3D-printed cardholders Each sample is labeled with card identities, zone assignments, and the projected 2D image coordinates for individual cards and playing field zones The data collection procedure simulates games between multiple artificial agents, simplifying the rules for ingredient card placement to create a greater variety of game states Each new game state is recorded as a sample consisting of an RGB image, the camera’s 6D pose and field of view, and for each card stand, card, and dummy point, as well as the playing field Another sample of the same game state is then recorded after randomizing the scene without affecting card placement Where the gains come from The primary gain comes from using domain randomization to bridge the reality gap, which makes models trained on synthetic data more robust to shifts in input domain by altering visual properties or dynamics The use of YOLO26n and YOLO26n-seg models finetuned with our randomized data achieved the highest F1-score of 0.906 This superior performance compared to models finetuned exclusively on canonical data shows that domain randomization greatly increases the zero-shot sim-to-real performance The combination of object detection and segmentation yields the best overall performance among the models evaluated The results show that both for the detection of cards and for the segmentation of game zones to assign those cards to, our domain randomization approach greatly increases the zero-shot sim-to-real performance of a model finetuned exclusively on synthetic data Pruning redundant reasoning tokens The authors report that all four evaluated object detection models finetuned with our domain randomization approach significantly outperform the model finetuned with canonical data only Overall, YOLO26n achieved the best object detection results with an F1-score of 0.845, an mAP50 of 0.850, and an mAP50:95 of 0.657 The YOLO26 models generally outperform the YOLO11 models, although the difference is much more evident with YOLO11n, while YOLO11s achieves comparable results Surprisingly, the increase in parameters between YOLO26n and YOLO26s yields worse results The combination of YOLO26n and YOLO26n-seg achieved the highest F1-score of 0.906, missing 1.52 cards on average Thus, our results show that both for the detection of cards and for the segmentation of game zones to assign those cards to, our domain randomization approach greatly increases the zero-shot sim-to-real performance of a model finetuned exclusively on synthetic data The baseline models finetuned exclusively on canonical data are unable to fully assess the game state with an F1-score of 0.627, with an average of 7.02 missing cards The results demonstrate that domain randomization is essential for the zeroshot sim-to-real transfer of our detection pipeline Considering the intentionally adverse lighting conditions in some of our test cases, this demonstrates the resilience of our model The authors conclude that domain randomization is essential for the zeroshot sim-to-real transfer of our detection pipeline We presented a fully automated synthetic data collection pipeline in CoppeliaSim for the Chef’s Hat card game, which automatically generates labels through reprojection and applies domain randomization to bridge the reality gap Using this data, we finetuned a YOLO26n object detection model to locate and classify individual cards and a YOLO26n-seg segmentation model to identify game-relevant zones of the playing field Our results demonstrate that domain randomization is essential for the zeroshot sim-to-real transfer of our detection pipeline The baseline models finetuned exclusively on canonical synthetic data are unable to fully assess the game state with an F1-score of 0.627, missing 7.02 cards on average In contrast, our domain-randomized models reduce the missing card average to 1.52 and achieve an F1-score of 0.

Title and authors: Rosa: Moving on, let's talk about the specific paper itself, "Reading the Table: Sim-to-Real Game State Detection for the Chef's Hat Card Game." The authors are Connor Gäde, Matthias Kerzel, Hauke Bünning, Jan-Gerrit Habekost, Mariana Barros, and Stefan Wermter from Knowledge Technology at the University of Hamburg.

Dev: It’s important to know who’s behind this work because they come from a strong background in knowledge technology and artificial intelligence systems.

Taro: I was looking at their related work on papers like MolmoB0T or ALAM, and this paper seems to be taking that idea of generating procedural data and applying it directly to a specific, tangible task like card manipulation.

Rosa: That’s right. The core idea they are proposing is using a fully automated synthetic data collection pipeline in CoppeliaSim1 to bypass the manual annotation headache entirely.

Dev: They also emphasize how this setup allows them to create a dataset of twenty-five thousand samples consisting of paired domain-randomized and non-randomized canonical images showing the Chef’s Hat playing cards on the playing field and custom three dimensional cardholders <ref:2611.15524#pg1>.

Taro: So, they aren't just making one kind of synthetic data; they are creating a variety that includes both perfect simulation setups and deliberately messy ones to test how good their models actually are.

Rosa: And that messy part is where the domain randomization comes in, which is designed specifically to bridge the reality gap when moving from simulation to the real world.

Dev: That randomization strategy involves altering visual parameters like lighting and background textures by moving four light sources up to two meters in either direction along two axes at a fixed height of two point five meters, oriented randomly.

Taro: They also randomize camera parameters, shifting the camera pose and field of view by moving it up to ten centimeters from its default position on all three axes and rotating it up to zero point two radians around all three axes.

Rosa: And they mention that the playing field and cardholders are also randomly moved and rotated on the table in their simulation to match how imprecise real-world placements actually are.

Dev: It’s a very comprehensive set of randomization techniques designed to stress test the perception model against unpredictable real-world variations.

Taro: It makes sense that they focus so heavily on this part, because if you can't handle those shifts, the model trained only on perfect simulation data just won't work when it sees something slightly different in reality.

Rosa: Exactly. And this whole pipeline is then used to finetune a combined object detection and segmentation model for assigning cards to zones and inferring the game state.

Dev: So, the next step is combining those two models—YOLO two six n for detection and YOLO two six n-seg for segmentation—to get that game state inference capability.

Taro: I wonder what happens when we look at the specific architecture choices they made, like choosing YOLO two six n over something else.

Rosa: Well, they found that the combination of YOLO two six n and the segmentation model gave them their best overall performance, achieving an F1 score of zero point nine zero six.

The paper's summary: Dev: So we’ve seen how they built this data pipeline and the randomization strategy, so now let’s look at what they actually found when they evaluated it. The summary of "Reading the Table: Sim-to-Real Game State Detection for the Chef's Hat Card Game" focuses on showing how their approach performs when tested on a real NICO humanoid robot.

Rosa: They are really demonstrating that this synthetic data collection pipeline, paired with domain randomization, provides a strong perceptual foundation for a robot to autonomously play Chef’s Hat against human players.

Taro: The key finding they highlight is the zero-shot sim-to-real performance improvement when using their randomized data versus models trained only on canonical synthetic data.

Dev: They report that baseline models finetuned only on canonical synthetic data were unable to fully assess the game state, missing seven point zero two cards on average.

Rosa: But with their domain-randomized models, that average missing card count drops down to one point five two and they hit an F1 score of zero point nine zero six.

Taro: So it’s not just a small bump; it shows that the randomization is essential for improving the model’s ability to interpret unseen real-world domains.

Dev: That means their method makes models trained on synthetic data much more robust to shifts in input domain by altering visual properties or dynamics.

Rosa: And this capability is what they are aiming for, which is integrating this perception pipeline with an embodied agent like ELMiRA to enable fully autonomous play of Chef’s Hat.

The paper's improvements: Taro: Now, the paper does suggest some improvements for future work beyond what they just achieved. They point out that relying solely on detection and segmentation might not be enough to capture the whole picture.

Dev: They explore exploiting game logic constraints to see if we can get a more accurate game state representation without needing perfect detection rates. They think this is an application of the frame axiom, meaning the game state can only change through player actions allowed by the game rules during their turn.

Rosa: That’s interesting because it suggests that even with a misdetection rate of approximately one point five cards, a single observation might not give a perfect representation of everything.

Dev: Exactly. But they argue that human players also don't capture the complete state with just one glance; they continuously update their internal representation while watching other players’ actions happen.

Taro: So, the paper suggests that an agent can focus on just a few cards played by other players and incrementally update the game state over time instead of needing a perfect single snapshot.

Rosa: That shifts the goal from finding perfection to finding a sufficiently accurate representation based on game logic, which is much more practical for real-world robotics.

Conclusion: Dev: So wrapping up this discussion on "Reading the Table: Sim-to-Real Game State Detection for the Chef's Hat Card Game," they’ve shown that domain randomization is crucial for zero-shot sim to real transfer.

Rosa: They proved that using their automated pipeline, which includes re-projection and domain randomization, significantly boosts model performance when tested on a real robot compared to models trained only on canonical data.

Taro: So the results show the baseline models missing about seven point zero two cards on average versus one point five two missing cards with the randomized approach.

Dev: The ultimate goal is integrating this perception pipeline with an embodied agent like ELMiRA so that we can achieve fully autonomous play of Chef’s Hat by combining game state inference with physical card manipulation and social interaction.

Rosa: It’s a lot of work, but it shows the path forward for how we can build robots capable of engaging in complex, socially interactive tasks.

Taro: I just hope they keep pushing on these kinds of scenarios where the environment is messy and unpredictable because that's where real robotics lives.

University of Hamburg

cs.RO, cs.CV

Submitted: 2026-11-11

Updated: 2026-11-11

Comments: Presented at the Dungeons, Neurons, and Dialogues workshop (DnD-SIDC 2026) at ICANN 2026, https://dnd-sidc.github.io/

Code: https://github.com/pablovin/ChefsHatGYM

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: Tabletop games offer a compelling scenario to study the interaction between artificial and human agents…

Key concepts

Domain Randomization
This technique intentionally changes visual and environmental aspects of the simulation, such as lighting colors and camera positions, to simulate different real-world conditions. This makes a trained AI model more robust and resilient when it encounters unseen environments outside the original training setup.
Object Detection and Segmentation
These are two AI tasks used together. Object detection finds and classifies individual cards in an image, while segmentation identifies the specific zones on the playing field where those cards belong. Combining them allows the system to accurately assign each detected card to its correct game area.
Sim-to-Real Transfer
This refers to the ability of an AI model trained entirely in a simulated environment (like CoppeliaSim) to perform well when deployed in a real physical world. The paper shows that domain randomization is crucial for making this transfer successful, allowing synthetic training data to effectively guide real-world performance.
Game State Inference
This is the process of determining the current status of the game based on visual input. The system achieves this by using object detection and segmentation to locate cards and then mapping those cards onto predefined game zones, thereby inferring which part of the game is currently active.

Terminology

Summary

Tabletop games offer a compelling scenario to study the interaction between artificial and human agents, combining social interaction with game state tracking and physical manipulation The authors developed a fully automated synthetic data collection pipeline using re-projection of known 3D coordinates to generate labels automatically and domain randomization to bridge the reality gap This approach was used to collect a dataset of 25413 samples to finetune an object detection and a segmentation model, which are combined to assign detected cards to game-relevant zones and infer the game state When evaluated on a real NICO humanoid robot, our pipeline achieves an F1-score of 0. p. 0906 for card-to-zone assignment, thus providing the perceptual foundation for a robot that can autonomously play Chef’s Hat against human players

Data Generation Pipeline

The synthetic data collection pipeline is fully automated in CoppeliaSim and creates a dataset of 25413 samples consisting of paired domain-randomized and non-randomized canonical images showing the Chef’s Hat playing cards on the playing field and custom 3D-printed cardholders Each image is labeled with card identities, zone assignments, and the projected 2D image coordinates for individual cards and playing field zones, from which bounding boxes and segmentation masks can be derived Each sample further includes the 6D simulation poses of all objects, camera parameters, and a random seed, allowing the reconstruction of any scene in simulation and enabling future extensions such as card pose estimation or motion planning for physical manipulation

Domain Randomization Strategy

Domain randomization is employed to mitigate the reality gap by altering visual and environmental parameters of the simulation to increase the resilience of a trained model to domain shifts and thus its ability to interpret the unseen real-world domain Specifically, lighting and background textures are randomized by moving four light sources up to 2 m in either direction along two axes at a fixed height of 2.5 m, oriented randomly, and having the color of both their specular and diffuse component randomized The camera pose and field of view are also randomized by moving the camera up to 10 cm from its default position in either direction on all three axes, rotating it up to 0. p. 62 radians in either direction around all three axes, and selecting a random field of view between 90° and 135° Furthermore, the playing field and cardholders are randomly moved and rotated on the table to reflect the changing, imprecise real-world placements

Game State Inference Mechanism

The game state detection pipeline divides the problem into two sub-tasks: Object detection to locate and classify individual cards in view, and segmentation to identify different game-relevant zones of the observation image that the cards have to be assigned to These tasks are solved by finetuning both the YOLO26n and YOLO26n-seg models from Ultralytics for object detection and instance segmentation on our domain-randomized dataset, using the canonical data as the validation set For each card detected by the object detection model, we take the center of the predicted bounding box and check whether it lies within the segmentation mask of one of the game-relevant zones predicted by the segmentation model, thereby enabling us to assign individual cards to game zones

Experimental Results and Conclusion

The evaluation on a real-world test set of 100 rectified images recorded from the left fisheye camera of a real NICO robot under 5 conditions varying in lighting and card placement shows that our domain randomization approach greatly increases the zero-shot sim-to-real performance of a model finetuned exclusively on synthetic data The baseline models finetuned exclusively on canonical synthetic data are unable to fully assess the game state with an F1-score of 0.627, missing 7. p. 9 p. 902 cards on average In contrast, our domain-randomized models reduce the missing card average to 1. p. 9 p. 952 and achieve an F1-score of 0. p. 0 p. 0906 This demonstrates the resilience of our model The ultimate goal is to integrate our perception pipeline with an embodied agent such as ELMiRA [4] to enable fully autonomous play of Chef’s Hat, combining game state inference with physical card manipulation and social interaction with human players

Pruning redundant reasoning tokens

The authors hypothesize that exploiting game logic constraints can lead to a sufficiently accurate game state representation in a real-world scenario without a perfect detection rate This is considered an application of the frame axiom: The game state can only change through player actions permitted by the game rules during their turn The misdetection rate of approximately 1.5 cards implies that a single observation may not provide a perfect representation of the game state However, human players are also unlikely to capture the complete game state with a single glance Instead, they continuously update their internal representation of the game while observing other players’ actions A robotic player can focus on the few cards played by other players at a time and incrementally update the game state

How the training signal is built

The dataset consists of 25413 samples consisting of paired domain-randomized and non-randomized canonical images showing the Chef’s Hat playing cards on the playing field and custom 3D-printed cardholders Each sample is labeled with card identities, zone assignments, and the projected 2D image coordinates for individual cards and playing field zones The data collection procedure simulates games between multiple artificial agents, simplifying the rules for ingredient card placement to create a greater variety of game states Each new game state is recorded as a sample consisting of an RGB image, the camera’s 6D pose and field of view, and for each card stand, card, and dummy point, as well as the playing field Another sample of the same game state is then recorded after randomizing the scene without affecting card placement

Where the gains come from

The primary gain comes from using domain randomization to bridge the reality gap, which makes models trained on synthetic data more robust to shifts in input domain by altering visual properties or dynamics The use of YOLO26n and YOLO26n-seg models finetuned with our randomized data achieved the highest F1-score of 0.906 This superior performance compared to models finetuned exclusively on canonical data shows that domain randomization greatly increases the zero-shot sim-to-real performance The combination of object detection and segmentation yields the best overall performance among the models evaluated The results show that both for the detection of cards and for the segmentation of game zones to assign those cards to, our domain randomization approach greatly increases the zero-shot sim-to-real performance of a model finetuned exclusively on synthetic data

The authors report that all four evaluated object detection models finetuned with our domain randomization approach significantly outperform the model finetuned with canonical data only Overall, YOLO26n achieved the best object detection results with an F1-score of 0.845, an mAP50 of 0. p. 8850, and an mAP50:95 of 0. p. 8657 The YOLO26 models generally outperform the YOLO11 models, although the difference is much more evident with YOLO11n, while YOLO11s achieves comparable results Surprisingly, the increase in parameters between YOLO26n and YOLO26s yields worse results The combination of YOLO26n and YOLO26n-seg achieved the highest F1-score of 0. p. 8906, missing 1. p. 952 cards on average Thus, our results show that both for the detection of cards and for the segmentation of game zones to assign those cards to, our domain randomization approach greatly increases the zero-shot sim-to-real performance of a model finetuned exclusively on synthetic data The baseline models finetuned exclusively on canonical data are unable to fully assess the game state with an F1-score of 0. p. 9627, with an average of 7. p. 902 missing cards The results demonstrate that domain randomization is essential for the zeroshot sim-to-real transfer of our detection pipeline Considering the intentionally adverse lighting conditions in some of our test cases, this demonstrates the resilience of our model The authors conclude that domain randomization is essential for the zeroshot sim-to-real transfer of our detection pipeline

Conclusion

We presented a fully automated synthetic data collection pipeline in CoppeliaSim for the Chef’s Hat card game, which automatically generates labels through reprojection and applies domain randomization to bridge the reality gap Using this data, we finetuned a YOLO26n object detection model to locate and classify individual cards and a YOLO26n-seg segmentation model to identify game-relevant zones of the playing field Our results demonstrate that domain randomization is essential for the zeroshot sim-to-real transfer of our detection pipeline The baseline models finetuned exclusively on canonical synthetic data are unable to fully assess the game state with an F1-score of 0.627, missing 7.02 cards on average In contrast, our domain-randomized models reduce the missing card average to 1.52 and achieve an F1-score of 0.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Automated synthetic data collection pipeline: Develop a fully automated synthetic data collection pipeline in CoppeliaSim1 that uses re-projection of known 3D coordinates to generate labels automatically and domain randomization to bridge the reality gap, creating a dataset of 25413 samples without the need for manual annotation.

  2. Game state detection inference: Create a game state detection approach that combines an object detection model (YOLO26n) and a segmentation model (YOLO26n-seg) to assign detected cards to game-relevant zones, enabling the system to assign the individual cards to the game zones.

  3. Robustness against domain shifts: Implement domain randomization techniques that visually alter scene parameters such as camera perspective, lighting, and background textures, as described in Section 3.3 and Section 7 of the paper, to increase the resilience of trained models to domain shifts and its ability to interpret the unseen real-world domain.

  4. Performance optimization through model selection: Evaluate different YOLO architectures (YOLO26n vs. YOLO26s) finetuned with synthetic data to determine which model variant yields the best results, as shown in Table 1, demonstrating that YOLO26n achieved the best object detection results with an F1-score of 0.845.

  5. Improved zero-shot sim-to-real transfer: Utilize domain randomization to improve the zero-shot sim-to-real performance of models finetuned exclusively on synthetic data, as this approach greatly increases the zero-shot sim-to-real performance of a model finetuned exclusively on synthetic data.

Sources

Related papers