ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarios

arXiv:2603.09741 · cs.CV · Submitted 2026-03-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarios".

Jane: Here's the gist of the paper. Understanding human behavior from complementary egocentric (ego) and exocentric (exo) points of view enables the development of systems that can support workers in industrial environments and enhance their safety <ref:2603.09741#pg2> The dataset is composed of 180 egocentric and 180 exocentric procedural videos temporally synchronized offering complementary information of the same scene <ref:2603.09741#pg3> These results highlight the need for new models capable of robust egoexo understanding in real-world environments Dataset Composition and Acquisition ENIGMA-360 is a new egoexo dataset acquired in a real industrial scenario The dataset is composed of 180 egocentric and 180 exocentric procedural videos temporally synchronized offering complementary information of the same scene It was acquired in a real industrial laboratory equipped with real industrial tools and components such as a power supply unit, soldering iron, and electrical boards The video pairs are temporally synchronized, with each pair capturing the same scene from two distinct perspectives To collect the dataset, they designed two realistic maintenance procedures to ensure the dataset reflects realistic industrial workflows They developed a wearable application that provides step-by-step instructions to participants during data collection The dataset is richly annotated to support a wide range of research tasks from both egocentric and exocentric perspectives Annotation Details The dataset includes temporal annotations marking the start and end times of keysteps, as well as spatial annotations identifying hand-object interactions within keyframes They provided segmentation masks for hands and manipulated objects using SAM-HQ to enhance the segmentation of complex objects They also released DINOv2 video features to extract rich spatial features from video frames Furthermore, they made available the 3D models of the laboratory and all industrial objects present in the dataset to support synthetic-to-real transfer Evaluation Tasks and Findings The authors performed baseline experiments for three foundational tasks: Temporal Action Segmentation, Keystep Recognition, and Egocentric Human-Object Interaction Detection For Temporal Action Segmentation, the FACT model achieved the best performance across all metrics when the source and target domains were the same In cross-domain scenarios, models like C2F-TCN consistently achieved the best and second-best results in both EGO→EXO and EXO→EGO settings For Keystep Recognition, the egocentric view consistently outperformed the exocentric view across all evaluation metrics, suggesting it provides richer and more detailed information In Egocentric Hand-Object Interaction Detection, the VISOR HOS baseline using segmentation masks achieved an Overall AP of 63.84%, which was enhanced by incorporating Label Propagation to 64.

Paper summary: Tom: So we’re talking about this paper called "ENIGMA-three hundred sixty: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarios" <ref:2603.09741#pg1,ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial>. Basically, it tackles a problem where we need AI to help workers in factories or workshops.

Jane: Right, so the main idea here is that understanding human actions from two different points of view—from the worker's perspective, called egocentric, and from an outside perspective, exocentric—is key to building systems that can actually support people and make them safer in these industrial settings.

Lu: The authors are addressing a real gap because most existing industrial datasets are either using toy assemblies or simplified environments, which just don't capture the visual complexity of real factory work three. They say they needed something that reflects actual industrial workflows.

Meng: I’m curious about what makes this dataset realistic. What kind of stuff did they actually put in there? Because if it only has toy tools, it won't help when you try to deploy a model on a real assembly line.

Tom: They acquired this thing in a real industrial laboratory, complete with actual tools like power supply units and soldering irons two <ref:2603.09741#pg1>. They synchronized one hundred eighty egocentric videos with one hundred eighty exocentric procedural videos, meaning both views are of the exact same scene at the exact same time one <ref:2603.09741#pg1>.

Jane: And that temporal synchronization is crucial because it means you get complementary information from both views, which is what they really claim enables a better understanding of behavior across those two perspectives three.

Lu: They designed two realistic maintenance procedures specifically to collect the data, and they used a wearable application to guide participants through the steps while recording one <ref:2603.09741#pg1>. This suggests a very controlled way of getting that procedural data.

Meng: That sounds like it took serious coordination. When you’re collecting real-world procedural data, how do you ensure the annotations are accurate across both views? I worry about noise if one view is messy and the other isn't one <ref:2603.09741#pg1>.

Tom: They did rich annotation work, which includes temporal annotations marking when each step starts and ends, plus spatial annotations identifying where hands are interacting with objects in keyframes one <ref:2603.09741#pg1>. They also used SAM-HQ for better hand segmentation on those complex objects.

Jane: And they didn't stop there; they released DINOv2 video features to pull out rich spatial information from the video frames themselves one <ref:2603.09741#pg1>. Plus, they even made the three dee models of the lab and all the industrial objects available, which helps with synthetic-to-real transfer later on one.

Lu: From a research standpoint, these annotations are powerful because they allow researchers to test a huge range of tasks from both egocentric and exocentric angles one <ref:2603.09741#pg1>. It gives them a lot of flexibility for different kinds of AI problems.

Paper summary: Meng: So, if you’re looking at deployment, how much effort does it take for someone to get ready with this data? Is it just collecting the videos, or is the annotation process itself very labor-intensive?

Tom: They focused on making the acquisition process realistic first so that they could build a dataset that actually matters for deployable models three. The goal was to move away from those simplified scenarios mentioned earlier in their work three.

Jane: So, when we look at the results they show, they tested three foundational tasks: Temporal Action Segmentation, Keystep Recognition, and Egocentric Human-Object Interaction Detection one <ref:2603.09741#pg1>. These are basically the core things they wanted to see if their dataset was good enough for.

Lu: For temporal action segmentation, their FACT model actually got the best performance when the source and target domains were identical one <ref:2603.09741#pg1>. But in cross-domain tests, models like C2F-TCN kept coming out on top in both ego exo and exo ego settings one <ref:2603.09741#pg1>.

Tom: Interesting. That suggests that for understanding actions across different perspectives, those specific model architectures are really robust even when the data isn't perfectly matched one <ref:2603.09741#pg1>. But what about identifying the actual steps?

Jane: For keystep recognition, they found that the egocentric view consistently performed better than the exocentric view across all metrics one <ref:2603.09741#pg1>. That’s a strong finding because it implies that seeing things from your own viewpoint gives you richer detail for recognizing specific actions.

Meng: So if I were building a system for monitoring maintenance, would I lean on egocentric data more or exocentric data? What does that difference actually mean for the kind of AI I’d be training?

Lu: It means the egocentric view provides more fine-grained information about what’s happening right there in front of the worker one <ref:2603.09741#pg1>. This level of detail is essential when you need to understand subtle hand-object interactions.

Tom: Speaking of interactions, they tested Egocentric Hand-Object Interaction Detection using a VISOR HOS baseline that used those segmentation masks we talked about earlier one <ref:2603.09741#pg1>. That baseline hit an overall accuracy of sixty-three point eight four percent, and they improved that by incorporating label propagation to sixty-four percent one.

Jane: Sixty-four percent is solid for an interaction detection task in a real setting, and it shows that even with the complexity of industrial tools, you can get decent results if you use good segmentation one <ref:2603.09741#pg1>. But what’s the bigger picture here?

Meng: For me, the big question is deployment. If I take this data and try to run a model on actual factory traffic at scale, what are the limitations? Does that sixty-four percent accuracy hold up when things get messy?

Paper summary: Lu: The paper did flag that because they focused on real industrial contexts, there are still challenges related to capturing data in truly complex settings like those with privacy regulations or proprietary processes three. They acknowledge that the complexity is still high.

Tom: So, the dataset is a step up from toy scenarios, but it’s not perfect for everything three. The key thing they deliver is this dual-view setup which provides that complementary information about the same scene one <ref:2603.09741#pg1>.

Jane: So to wrap up this part of the paper on ENIGMA-three hundred sixty it’s really about creating a high-quality dataset that captures how workers actually operate in real industrial environments by linking two synchronized views one <ref:2603.09741#pg1>. It sets a new standard for what we expect from ego exo understanding in these challenging scenarios.

Lu: And the implication is that future AI agents meant to support physical work need to be trained on this kind of rich, multi-view data to function reliably one <ref:2603.09741#pg1>. This moves us closer to systems that can truly understand complex procedures without relying on simplified models.

Meng: I think it means we need more data like this because the current benchmarks don't simulate the real variability of a factory floor well enough for production deployment three. If you want an AI that actually works on a live line, you need data that breaks at scale, not just in a lab setting.

Tom: Exactly. The work on ENIGMA-three hundred sixty shows the direction we need to go: moving from simulated or toy environments toward real-world industrial scenarios where both ego and exo information are available one <ref:2603.09741#pg1>. That’s where the useful models will be built.

Jane: So, what does this mean for us listening right now, maybe if you’re thinking about how AI might help your own work environment? It means we need to think about the context of where that AI is being used—is it viewing it from inside or outside?

Lu: And from a research angle, the focus on temporal and spatial annotations across these views gives us new tools for representation learning that should eventually apply to other complex human-machine interaction tasks one <ref:2603.09741#pg1>. It’s about learning how to see the world through different lenses.

Meng: For production engineers, it means when we look at building safety features, we need to consider both the worker's view and an external monitor's view simultaneously for a better risk assessment three. That’s a practical application of this kind of data collection.

Tom: The paper establishes that this dual-view approach is necessary for robust understanding, especially when dealing with complex procedures that involve many different tools and movements one <ref:2603.09741#pg1>. It’s about getting the full picture.

Jane: So, ENIGMA-three hundred sixty is positioning itself as a crucial resource because it bridges the gap between controlled lab data and the messy reality of industrial work one <ref:2603.09741#pg1>. It shows what we need to collect to build truly helpful industrial assistance systems.

Conclusion: Tom: So we’re wrapping up our look at ENIGMA-three hundred sixty which is this new dataset for understanding human behavior in industrial settings by using both egocentric and exocentric views.

Jane: Right, so essentially they've put together a massive collection of synchronized videos from a real factory floor to give AI richer information about how people work.

Lu: The authors built this set from eighteen zero egocentric videos and eighteen zero exocentric procedural videos, all captured in a genuine industrial lab setting with actual tools.

Meng: So it’s not some clean simulation; it’s real stuff, like power supply units and soldering irons, which means the data should reflect actual complexity.

Tom: Exactly. And they did this so researchers can test models on these dual perspectives, which is a big deal for making AI that actually works in the real world.

Jane: They also released all the annotation details, like temporal markers for each step and spatial maps showing where hands were interacting with objects in keyframes.

Lu: Plus they provided DINOv2 video features and even three dee models of the lab and all those industrial objects to help with transferring what they learn to new situations.

Meng: From a deployment standpoint, having that kind of rich annotation means you’re not starting from scratch when trying to build a safety system or a monitoring tool.

Tom: It really highlights that for complex tasks, just one way of looking at things isn't enough; you need both the worker's view and an external view to get the full picture.

Jane: So what does this mean for how we think about industrial AI? It suggests that future systems meant to help people in factories need to be trained on this kind of multi-view data.

Lu: The authors are essentially saying that moving away from toy environments toward these real-world scenarios is where the next big step for robust ego exo understanding happens.

Meng: But they did mention there are still challenges, especially with things like privacy and keeping up with the sheer messiness of a live factory floor.

Tom: True. So ENIGMA-three hundred sixty gives us a really high-quality foundation, but we still have to figure out how to handle that real-world noise when we try to scale these models up.

University of Catania

cs.CV

Submitted: 2026-03-10

Updated: 2026-11-12

Code: https://github.com/pndurette/gTTS

Project page: https://fpv-iplab.github.io/ENIGMA-360

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 77/100

The gist: Understanding human behavior from complementary egocentric (ego) and exocentric (exo) points of view enables the development of systems that can support workers in industrial environments and enhance…

Key concepts

Egocentric View
This perspective views the scene from the point of view of a person performing an action. It captures what the worker sees and does during a task, offering a direct, first-person understanding of their movements and interactions with tools.
Exocentric View
This perspective captures the same scene from an external viewpoint, like a fixed camera in the environment. It provides context about the overall workspace and how the worker's actions fit into the larger industrial setting.
Temporal Synchronization
The dataset pairs videos that capture identical scenes at specific moments in time from both egocentric and exocentric angles. This synchronization is vital because it allows researchers to compare what a person sees versus what an external observer sees during the exact same keystep or action.
Keystep Recognition
This task involves identifying the specific steps taken during a procedure. The study found that egocentric views were better at recognizing these steps because they provide richer, more detailed information about the precise actions being performed.

Terminology

Summary

Understanding human behavior from complementary egocentric (ego) and exocentric (exo) points of view enables the development of systems that can support workers in industrial environments and enhance their safety <ref:2603. p. 0

Where the paper says it

ragusa@unict. it; Abstract Understanding human behavior from complementary egocentric (ego) and exocentric (exo) points of view enables the development of systems that can support workers in industrial environments and enhance their safety.

09741#pg2> The dataset is composed of 180 egocentric and 180 exocentric procedural videos temporally synchronized offering complementary information of the same scene <ref:2603. p. 0
Where the paper says it

To address this gap, we propose ENIGMA-360, a new egoexo dataset acquired in a real industrial scenario. The dataset is composed of 180 egocentric and 180 exocentric procedural videos temporally synchronized offering complementary information of the same scene.

09741#pg3> These results highlight the need for new models capable of robust egoexo understanding in real-world environments

Dataset Composition and Acquisition

ENIGMA-360 is a new egoexo dataset acquired in a real industrial scenario The dataset is composed of 180 egocentric and 180 exocentric procedural videos temporally synchronized offering complementary information of the same scene It was acquired in a real industrial laboratory equipped with real industrial tools and components such as a power supply unit, soldering iron, and electrical boards The video pairs are temporally synchronized, with each pair capturing the same scene from two distinct perspectives To collect the dataset, they designed two realistic maintenance procedures to ensure the dataset reflects realistic industrial workflows They developed a wearable application that provides step-by-step instructions to participants during data collection The dataset is richly annotated to support a wide range of research tasks from both egocentric and exocentric perspectives

Annotation Details

The dataset includes temporal annotations marking the start and end times of keysteps, as well as spatial annotations identifying hand-object interactions within keyframes They provided segmentation masks for hands and manipulated objects using SAM-HQ to enhance the segmentation of complex objects They also released DINOv2 video features to extract rich spatial features from video frames Furthermore, they made available the 3D models of the laboratory and all industrial objects present in the dataset to support synthetic-to-real transfer

Evaluation Tasks and Findings

The authors performed baseline experiments for three foundational tasks: Temporal Action Segmentation, Keystep Recognition, and Egocentric Human-Object Interaction Detection For Temporal Action Segmentation, the FACT model achieved the best performance across all metrics when the source and target domains were the same In cross-domain scenarios, models like C2F-TCN consistently achieved the best and second-best results in both EGO→EXO and EXO→EGO settings For Keystep Recognition, the egocentric view consistently outperformed the exocentric view across all evaluation metrics, suggesting it provides richer and more detailed information In Egocentric Hand-Object Interaction Detection, the VISOR HOS baseline using segmentation masks achieved an Overall AP of 63.84%, which was enhanced by incorporating Label Propagation to 64. p. 2

Where the paper says it

The performance gap observed across methods highlights the substantial domain gap between industrial and everyday activities and confirms that ENIGMA-360 poses new challenges for human behavior understanding. This work extends our previous research [19] by incorporating 129 additional egocentric videos and 180 exocentric videos, enriching the dataset with new temporal and spatial annotations, and introducing a benchmark tailored to the analysis of human behavior in industrial environments.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Develop models capable of robust egoexo understanding in real-world environments by leveraging ENIGMA-360, which consists of 180 egocentric and 180 exocentric procedural videos temporally synchronized offering complementary information of the same scene. This enables systems to provide holistic understanding of the same scene during industrial task execution.

  2. Implement Temporal Action Segmentation (TAS) systems using models like FACT or C2F-TCN, which can be trained on both egocentric and exocentric subsets to achieve cross-view generalization. This allows the system to accurately identify when specific actions begin and end is essential for assessing procedural correctness.

  3. Create Keystep Recognition systems that predict the keystep label kt for each temporal window t in a video V, focusing on identifying essential sub-events that define the structure of complex tasks. This supports downstream systems in reasoning about task progression at a fine-grained level by validating that mandatory steps are executed correctly and in sequence.

  4. Design Egocentric Hand-Object Interaction Detection systems using segmentation masks like VISOR HOS or Label Propagation methods to predict triplets including hand, hand contact state, active object. This allows the system to monitor operator performance by accurately characterizing contact patterns, manipulation strategies, and tool usage in industrial settings.

Abstract

Understanding human behavior from complementary egocentric (ego) and exocentric (exo) points of view enables the development of systems that can support workers in industrial environments and enhance their safety. However, progress in this area is hindered by the lack of datasets capturing both views in realistic industrial scenarios. To address this gap, we propose ENIGMA-360, a new ego-exo dataset acquired in a real industrial scenario. The dataset is composed of 180 egocentric and 180 exocentric procedural videos temporally synchronized offering complementary information of the same scene. The 360 videos have been labeled with temporal and spatial annotations, enabling the study of different aspects of human behavior in industrial domain. We provide baseline experiments for 4 foundational tasks for human behavior understanding: 1) Temporal Action Segmentation, 2) Keystep Recognition, 3) Egocentric Human-Object Interaction Detection, and 4) Video-Grounded Cross-View VQA, showing the limits of state-of-the-art approaches on this challenging scenario. These results highlight the need for new models capable of robust ego-exo understanding in real-world environments. We publicly release the dataset and its annotations at https://fpv-iplab.github.io/ENIGMA-360/.

Sources

Related papers