EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing

summary

The gist

We introduce a three-wave, in-the-wild multimodal dataset designed for affect sensing, integrating smartphone sensing, wearable sensing…

In short

This paper introduces EmoPhone, a three-wave dataset combining smartphone and wearable sensing with experience-sampling labels from 2020-2022. It provides longitudinal data allowing researchers to test how well models can predict affect in real-world mobile settings across different user behaviors and time periods. The findings highlight that user variability is a major challenge for robust affective computing.

Key concepts

Three-Wave Dataset
EmoPhone consists of three distinct, longitudinal waves (D-1, D-2, D-3) collected annually from university students. Each wave uses the same core sensing methods but has different design choices, allowing researchers to study how models generalize across these variations.
Experience-Sampling Method (ESM)
This method involves participants logging their affective states frequently over several weeks. The dataset includes dense labels derived from this sampling, providing rich, moment-level ground truth for affect sensing during active days.
Cross-Wave Generalization
This tests a model's ability to perform well when trained on one wave (e.g., D-1) and applied to another wave (e.g., D-3). The study found that domain generalization methods performed best here, suggesting models need to handle significant differences between deployment settings.
Within-User Temporal Prediction
This involves training a model on one user's history to predict their affective state within the next 30 days. This capability helps create a personalized reference point for forecasting future moods based on past self-observations.

Terminology used across episodes

This episode discusses

The paper

EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing · Read on arXiv

Graduate School of Data Science, KAIST, Republic of Korea · Samsung Electronics Co., Ltd., Republic of Korea · School of Computing, KAIST, Republic of Korea · R&D Department, HumbleBeeAI, Republic of Korea · Department of Data Science and Department of Computer Engineering, Kangwon National University, Republic of Korea · Department of Computer Engineering, Gachon University, Republic of Korea · Department of Computer Science and Information Systems, BITS Pilani K K Birla Goa Campus

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing".

Jane: Here's the gist of the paper. We introduce a three-wave, in-the-wild multimodal dataset designed for affect sensing, integrating smartphone sensing, wearable sensing, and dense experience-sampling method (ESM) labels collected annually from 2020 to 2022.

Tom: The resource comprises three longitudinal datasets: D-1 (102 participants), D-2 (112 participants), and D-3 (114 participants).

Jane: Each wave involved university students over approximately four weeks, combining repeated experience-sampling labels with passive sensing from their own Android phones and wearable devices.

Tom: The dataset is substantial in both participant count and label volume, yielding ESM totals of 10,259 for D-1, 21,042 for D-2, and 21,838 for D-3.

Title and authors: Tom: So Jane, we're looking at this paper now, "EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing." It sounds like they've put together a really substantial resource here for studying how people feel in real life.

Jane: Right. It’s a three-wave dataset, so they’ve collected data from two thousand twenty to two thousand twenty-two integrating phone sensing, wearable sensors, and experience sampling method labels <ref:2609.16581#pg1>. That integration across time and different devices is what makes it interesting for studying affect.

Lu: The core idea here is creating this structured benchmark in one resource so researchers can test models against something real that reflects the dynamic nature of affective states, which are short-horizon and person-dependent, you know?

Meng: I'm curious about the scale. They’re talking about three waves with one hundred two to one hundred fourteen participants each <ref:2609.16581#pg3>. That’s a decent participant count for longitudinal studies, but how much label data are we actually looking at across those waves?

Tom: Well, the ESM totals are pretty big. For D-one they have ten thousand two hundred fifty nine experience sampling method labels collected over the year <ref:2609.16581#pg3>. D-two brings that up to twenty-one thousand forty-two labels, and then D-three has twenty-one thousand eight hundred thirty-eight of those as well <ref:2609.16581#pg1>.

Jane: And they’ve made sure to keep the core affective labels—valence, arousal, stress, and disturbance—consistent across all three waves so you can compare things directly. But D-three adds extra descriptors like Happy, Relaxed, Cheerful, Content, Sad, Angry and more for a richer analysis <ref:2609.16581#pg1>.

Lu: That expansion in D-three is important because it gives you that finer granularity for emotion analysis beyond the basic core space that the other waves share <ref:2609.16581#pg1>. It’s about adding semantic depth to what you’re measuring.

Meng: From an engineering standpoint, I see them including keystroke sensing starting from D-two which adds another dimension to the input data we feed into models <ref:2609.16581#pg1>. That makes sense for capturing more behavioral context alongside the physiological and self-report data.

Tom: Exactly, it’s about gathering more signals to build a more robust representation of affect. But what I find really compelling is how they structured the evaluation around different prediction tasks.

Jane: They test three main things: predicting your future mood within thirty days using your own history, testing if one user’s model can predict another user in the same wave, and then testing cross-wave generalization across all three waves treated as separate datasets.

Title and authors: Lu: That separation is key because it directly addresses how robust a model actually is when it encounters a new person or a new year of data entirely. It forces you to think about the source of variability.

Meng: When I look at this, I wonder about the operational cost. They mention that D-three involves more complex descriptors, which might mean higher computational needs for those specific models compared to just using valence and arousal on D-one <ref:2609.16581#pg1>.

Tom: True, but they've benchmarked it against supervised baselines doing better in the temporal prediction setting, which suggests that when you’re predicting your own future state based on your history, a well-trained model really shines.

Jane: That makes sense because temporal prediction is essentially learning the trajectory of an individual user’s affect over time, which leverages their specific patterns. But what happens when we look at generalizing across users within the same wave?

Lu: The results here suggest that unsupervised domain adaptation methods are strongest overall in that within-wave cross-user setting, meaning adapting a model trained on one person to predict another person's behavior is where the representation learning really pays off.

Meng: That’s interesting because we’re always worried about bias when we generalize across users, and if UDA handles that well on the core labels like valence and arousal, it lowers the barrier for deploying these kinds of personalized systems.

Tom: But then there’s cross-wave generalization, treating each wave as a separate dataset entirely. The benchmark results show that domain generalization shows the strongest performance in this setting compared to the other two scenarios we discussed.

Jane: That means when we have to predict someone in D-three using only data from D-one or D-two treating them as a completely new domain is the most effective approach for that specific task <ref:2609.16581#pg1>. It tells us about how much cross-wave differences truly matter versus just individual user noise.

Lu: The bottleneck they pinpoint isn't just the label density itself, but this combination of substantial participant variability and those realistic cross-wave differences that come up in multi-year deployments. That’s a big warning for the future of this kind of computing.

Title and authors: Meng: If that variability is what matters most, then scaling up data volume might not solve our problems; we need better models to handle the inherent shift between years and users. What does that mean for deployment?

Tom: It means we can’t just train a model once and expect it to work perfectly across different cohorts or time periods without some form of adaptation mechanism built into the system. The challenge is managing that variability.

Jane: So, to summarize, this EmoPhone dataset gives us a longitudinal view of affect using multiple sensors and labels, showing us that while temporal prediction is strong within a user, cross-wave generalization requires specific domain adaptation or generalization techniques to handle the real-world shifts between deployments.

Lu: And they've shown that focusing on unsupervised domain adaptation for cross-user shifts gives the strongest results under those specific conditions. It’s a clear direction for representation learning work in this area.

Meng: For practical deployment, it highlights that building systems that can robustly handle these cross-wave differences is more critical than just having the most detailed label set available at any single moment.

Tom: So, to wrap up on EmoPhone: it’s a multi-year dataset combining phone and wearable data with experience sampling labels across three waves, showing that generalization depends heavily on whether you’re predicting within a user, between users in the same wave, or across different years entirely.

Jane: It really frames how we should think about building systems for moment-level affect prediction in the wild by acknowledging all these layers of variability. That’s a lot to take in for someone just listening to the show.

Lu: And it opens up avenues for developing better representation learning methods that can effectively bridge those gaps between different user populations and different years of data.

Meng: I’m just thinking about the operational side—if we want these models to be used in real-time, they have to be lightweight enough to handle this complexity without massive inference costs.

Tom: That’s the next big hurdle for anyone working on deploying this kind of multimodal affect sensing technology. The data is rich, but the computational demands need careful management as we move toward real-world applications.

Jane: Well, that gives us a good starting point for thinking about what to study next in affective computing research. We’ve got a lot of ground covered on EmoPhone today.

The paper's summary: Tom: So, EmoPhone is basically this big collection of data from three different years of people using their phones and wearables to track how they’re feeling in real life.

Jane: Right, it's like we have these three separate snapshots of human emotion—one year, then the next—and they put phone sensors and wearable stuff right into it for each snapshot.

Lu: The whole point is to create a standardized benchmark so researchers can test how well their AI models actually work when they’re trying to understand affect in the wild.

Meng: But what makes this resource interesting is how much real-world noise and user difference there is across those three years and participant groups.

Tom: Yeah, the authors found that just having a lot of labels isn't enough; the massive differences between users and between years make it a tough test for any system.

Jane: They set up three main ways to test models: predicting your own future feelings over thirty days using your past data, trying to predict another person in the same group, or testing if a model trained on one year of data can predict someone in a completely different year.

Lu: And their findings point toward specific methods being better for specific tasks; unsupervised domain adaptation works best when you're generalizing between users within the same wave.

Tom: But when it comes to jumping across those three waves, treating each year as totally separate, that's where they found that domain generalization methods actually perform the strongest overall on average.

Jane: Which tells us that for long-term affective computing in this setting, we need models that can handle both individual user shifts and big temporal changes between different collection periods.

Meng: From a deployment standpoint, this means any system we build has to be robust enough to handle not just the label density but also those inherent, messy differences between users and time periods.

Tom: Exactly. The main takeaway is that the bottleneck isn't just getting more labels; it’s managing this huge variability in how people act and how their feelings change over time.

Jane: It’s a lot to take in for someone just listening to the show, but it frames exactly what kind of challenges we need to focus on when building systems that sense human emotion outside a lab.

Lu: And that sets us up perfectly for looking at how those models actually learn these patterns and what specific architecture changes would help them adapt better to those cross-wave shifts.

The paper's improvements: Tom: So the paper isn't just about presenting the data; they’re suggesting specific ways to improve how we use this dataset for building AI models.

Jane: Right, they're basically saying that since real life is so messy and variable, we need better tools for adapting those models to work across different users and different years.

Lu: They propose focusing on unsupervised domain adaptation methods specifically when you’re trying to generalize predictions between users in the same wave.

Meng: That makes sense because if you're just shifting from one person to another in the same year, you don't need a massive labeled dataset for that; you just need the right way to adjust your model weights.

Tom: And they also suggest using those within-user temporal predictions—predicting your own future feelings based on what happened earlier—to create a personal reference point for each person.

Jane: That’s really smart because it lets the model learn an individual's specific pattern of change, which helps it forecast their next state more accurately than just looking at average behavior.

Lu: And they also look at adding those extra emotion descriptors, like Happy or Content from the D-three wave, to give models a richer semantic layer for finer analysis.

Meng: That’s useful for getting specific insights beyond just valence and arousal, but I still have to wonder if adding more labels just makes the training process slower without giving us a significant lift in actual deployment performance.

Tom: The authors acknowledge that the temporal prediction setup using your own history actually gets the best results when you look at supervised baselines because it’s a very direct way to measure prediction accuracy.

Jane: So, they’re suggesting a mix: using supervised learning for personal forecasting and unsupervised methods for handling the big shifts between different user groups or years.

Lu: It feels like they're giving us a blueprint for how to structure our modeling efforts so we aren't just looking at one single prediction task but considering the whole spectrum of generalization challenges.

Meng: I still worry about the computational cost of running these more complex adaptation methods on real-time mobile hardware, though I see the theoretical benefit for robustness.

Tom: That’s a fair point about deployment costs, but if we can get a model that performs reliably across those waves, the operational savings from not needing to constantly retrain for every new user might balance things out.

Jane: It really shows that the future of this kind of sensing isn't just about collecting more data; it's about building smarter systems that can intelligently bridge those gaps between different real-world conditions.

Conclusion: Tom: So to wrap up, EmoPhone is this three-wave dataset combining phone and wearable data with experience sampling labels across two thousand twenty to two thousand twenty-two for studying affect in the wild.

Jane: Basically, it shows how we can create a longitudinal view of human emotion by mixing different collection methods over several years.

Lu: It really demonstrates that when you’re studying something like affect, you can’t just rely on one snapshot; you need these multi-wave resources to understand the real movement.

Meng: The main implication is that for any AI system we build that needs to understand mood, it has to be designed with this kind of variability in mind from the start.

Tom: It forces us to think about how models handle shifts between users and shifts over time, not just label density alone.

Jane: It sets a high bar for what we consider a robust dataset when you’re trying to predict something as complex as human affect in real-world, messy situations.

Lu: And it gives us a solid framework for testing if unsupervised adaptation techniques can actually handle those cross-wave differences effectively.

Meng: I still think the operational challenge remains how fast we can get a model that adapts this well and runs on a device without needing massive computational power.

Tom: That’s the next big hurdle for anyone working on deploying this kind of multimodal sensing technology, making it lightweight enough to handle all that complexity in real-time.

Jane: It's definitely a resource we need to look at closely if we want to move beyond just single-moment predictions toward truly long-term affective computing.

Lu: So, moving forward, I’m excited to see how researchers apply these adaptation strategies we discussed to other challenging multimodal sensing tasks.

Meng: Yeah, and I’m going to be watching how those models scale when they try to predict affect for a whole new cohort of people who never appeared in the training data.

Lalam: I’m confident that EmoPhone provides a very clear path forward for making AI systems that sense human emotion more reliable across different contexts.

More episodes

← Home