Smiling Regulates Emotion During Traumatic Recollection

arXiv:2604.19019 · cs.MM · Submitted 2026-04-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Smiling Regulates Emotion During Traumatic Recollection".

Jane: Here's the gist of the paper. Smiling plays a critical role in regulating emotion and social interaction during traumatic recollection The study investigates when, where, and why 978 Holocaust survivors smile in video testimonies.

Tom: Study Objectives The research aimed to create an automatic smile detection model from facial features with an F1 of 85% and annotate detected smiles under two established taxonomies of smiling.

Jane: The study also produced narrative features on 1,083,417 transcript sentences as well as emotional valence from three different modalities: audio, eye gaze, and text transcript.

Tom: Smiling rates were associated with specific semantic topics, narrative structures, and temporal syntaxes across the corpus.

Paper summary: Tom: So we're looking at this paper today, "Smiling Regulates Emotion During Traumatic Recollection," and it’s about how smiling actually plays a role in managing feelings and social interactions when people are remembering traumatic events.

Jane: That sounds really important because trauma is such a heavy thing to process, right? So what is the main idea here? What are they claiming about smiles in these videos of survivors?

Lu: The paper sets out to create an automatic model that can detect smiles from facial features with an F1 score of eighty-five percent, and then they label those detected smiles using two different ways to categorize them: one based on social function, and another based on how authentic the feeling behind the smile is.

Meng: Okay, so they are building a detection model first, which sounds like standard machine learning stuff. But what's the big payoff for us as people listening? What do these smiles actually tell us?

Lalam: The study produces narrative features from over one million sentences and emotional valence data from three different ways: audio, eye gaze, and text transcript. And they found that negative-affect smiles are linked to more positive feeling trajectories across all three of those ways.

Tom: Negative-affect smiles leading to more positive feelings—that’s a specific link they’re pointing out there. So when we see someone smile during a memory of something bad, it might actually be signaling something different than you'd think.

Jane: Exactly. It suggests that smiling isn't just random; it seems to be part of the way people regulate emotion when they are recalling those difficult times, which is what the title is about.

Lu: They looked at how smiles relate to specific topics in the story, like narrative structures and temporal syntaxes—which is just a fancy way of saying how sentences are put together over time. They found that smiling significantly reduces the rate of orientation sentences and sentences set in the strict past by forty-seven percent and sixty-one percent respectively.

Meng: Forty-seven percent reduction in those types of sentences—that’s a big chunk of narrative structure changing just because someone is smiling while recalling something traumatic. That’s what I need to know for deployment; how does that translate into a usable signal?

Lalam: The paper also showed that smiling is associated with differences in eye dynamics and blink rates, and these changes vary depending on the narrative valence, which means whether the story part they are looking at is positive or negative.

Paper summary: Tom: So we have smiles linked to specific narrative structures—like reducing sentences about where things are or when they happened—and also linked to how eyes move during those moments. That sounds like a complex feedback loop between what’s being said and how the person looks while saying it.

Jane: It really emphasizes that smiling acts as a mediator, bridging the internal emotion with the social interaction happening between the survivor and whoever is recording them, which is why they call it regulation during recollection.

Lu: The authors conclude that smiling plays a critical role in regulating emotion and social interaction during traumatic recollection because it’s associated with specific semantic topics, narrative structures, and temporal syntaxes across the corpus.

Meng: So we have these findings about how smiles change the way narratives are structured and how they affect emotional feeling, but what does this mean for someone just listening to a story without knowing all that technical jargon?

Jane: It means that when you’re hearing a difficult memory, the way someone chooses to smile—whether it's polite or something more authentic—tells you a lot about what they are actually feeling underneath the surface.

Tom: And we have to remember the context here: almost a fifth of all smiles in their data occurred during moments of negative valence, which means even in trauma, smiling isn't just a simple happy signal.

Lalam: The paper also notes that this smiling is often a carefully constructed display of social expression that tries to balance internal affect with the emotions one wants to show the world.

Lu: That idea—that it’s a constructed display—is interesting because it brings in the idea of learned behavior, influenced by cultural backgrounds and personal context, which is something we need to factor into any model we build for this.

Meng: If we were trying to build a system to analyze social dynamics during stress, knowing that the smile itself has a social function, like masking negative emotion or being polite for agreement, that’s crucial information for deployment.

Jane: It gives us a framework for understanding not just *what* people are saying, but *how* they are interacting with their emotions in real time during something really hard.

Tom: The paper also mentions the authors used generative AI to help with coding and building that annotation tool, which is a cool detail about how modern researchers use tools to handle this kind of complex data.

Paper summary: Lu: The study’s limitation, which the authors themselves flagged, is that they are limited to analyzing only English-speaking subjects who survived internment at various concentration camps. So we can’t generalize those specific findings easily across all trauma experiences globally right now.

Meng: That makes sense for a specific dataset, but if we want to make a model that works on real-world data from different countries, how much do we have to change about the feature engineering based on these results?

Lalam: The most predictive features they found for just detecting a smile were the per-subject z-scores of facial action units AU06, which is cheek raising, and AU12, which is lip corner pulling. Those two alone achieved AUCs of zero point eight zero nine and zero point seven five six respectively.

Tom: So those specific muscle movements are the strongest indicators for a binary smile detection before you even look at the whole narrative context. That gives us a concrete starting point for our model training, I like that specificity.

Jane: It grounds the abstract concept of "emotion regulation" into tangible facial muscle movements that we can actually measure and process with an AI model.

Lu: And they used a combination of Action Units plus audio emotion for their best final smile detection model, which achieved an F1 score of zero point eight five. That tells us that combining visual cues with vocal tone gives them the most accurate result on this specific task.

Meng: If we deploy that on a real video feed, running a model that combines facial action units and audio signals sounds doable, but what’s the latency like? How fast does it have to process those inputs for this to be useful in a live setting?

Tom: The paper didn't give us the exact latency numbers, but they focused on training models with different architectures like MLP, GRU, and CNN over six temporal windows around the smile. That suggests they were testing how deep the model needed to look into the video stream.

Jane: So while we know *what* features are important for detection—AU06 and AU12—the real complexity lies in how those features interact with the surrounding conversation and narrative structure, which is what Lu’s work on representation learning is about.

Lu: Precisely, the paper shows that smiles aren't just isolated events; they are woven into specific patterns of sentences and temporal syntax across different modalities like audio, eye gaze, and text transcript. That requires a model that can handle all three at once.

Paper summary: Meng: For production, running three separate emotion detectors—one for audio valence, one for eye gaze VAD values from the previous five seconds, and one for text semantic features—that sounds like it could be computationally expensive to run continuously on low-power devices.

Lalam: The study's main limitation is that they focused their analysis on a subset of nine hundred seventy-eight English-speaking survivors, so we can’t assume this exact relationship holds true for every single traumatic recollection across every culture or language without further testing <ref:2604.19019#pg1>.

Tom: It’s a strong finding, though. It moves the conversation past just looking at the raw emotion and starts looking at the specific social and structural patterns that accompany it in these memories.

Jane: And what this means for people who only listen to these kinds of studies is that smiling isn't just a surface expression; it’s a complex mechanism that helps manage how we process deep, painful experiences like trauma.

Lu: It opens up avenues for research into how we can build representations of social interaction during stress that are sensitive to these subtle facial cues and narrative dynamics.

Meng: If I were building this on real traffic, I’d be worried about the noise—the background chatter—interfering with detecting those subtle eye gaze changes or the tiny muscle movements in the face.

Tom: So, to summarize for our listeners today: smiling during traumatic memory isn't just a mood indicator; it's linked to specific sentence structures and eye dynamics, and negative smiles actually precede more positive feelings across different ways we can measure emotion.

Jane: That’s the core of this paper on "Smiling Regulates Emotion During Traumatic Recollection," showing that these small expressions are actually part of a larger system for regulating how we handle difficult memories.

Lu: The implication is that future models need to look beyond just detecting a smile, but understanding how that specific smile fits into the entire narrative flow and the emotional context of what is being recalled.

Meng: And for those of us building things, it means we can start prioritizing features like AU06 and AU12 when designing our emotion detection pipelines if we want to capture that foundational signal reliably.

Lalam: We're really confident this finding holds up because they tied the smile detection to established taxonomies of both social function and emotional authenticity, giving us a solid way to interpret what those detected smiles mean in practice.

Conclusion: Tom: So, this paper is about how smiling actually helps control your emotions when you’re remembering something traumatic like the Holocaust or other genocides.

Jane: It looks at video testimonies and tries to figure out when and why people smile in those moments because it seems like a really complex thing to study.

Tom: Yeah, the authors set out to build a model that could automatically spot those smiles from facial features with eighty-five percent accuracy.

Jane: And they didn't just stop there; they looked at how smiling relates to different parts of the story, like what kind of sentences people use or how their eyes move.

Tom: They found that negative feelings accompanied by a smile actually lead to more positive feelings later on across all those different ways they measured emotion.

Jane: So the main idea is that smiling isn't just random; it’s actually part of a system where people regulate their emotions while recalling something terrible.

Tom: It suggests that these smiles are tied to specific topics in the story and how time moves through the narrative structure.

Jane: And what this means for us is that we can start looking at smiles not just as an emotion itself, but as a signal about how someone is managing their feelings in a very difficult situation.

Tom: It’s showing that smiling acts like a mediator between what's happening inside the person and how they interact with others during those painful memories.

Jane: That leads us to think about how we can use this kind of analysis to better understand social dynamics during intense stress or memory recall.

University of Southern California · Freie über University Berlin

cs.MM

Submitted: 2026-04-21

Updated: 2026-11-10

Comments: Received Best Paper Runner-Up Award at ICMI 2026

Journal ref: In Proceedings of the 28th ACM International Conference on Multimodal Interaction (ICMI '26), October 5-9, 2026, Napoli, Italy

DOI: 10.1145/3776574.3831180

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 78/100

The gist: Smiling plays a critical role in regulating emotion and social interaction during traumatic recollection The study investigates when, where, and why 978 Holocaust survivors smile in video testimonies.

Key concepts

Automatic Smile Detection Model
The study created an automated system using facial features to detect smiles in videos. It achieved high accuracy (F1 of 85%) by analyzing specific facial movements, such as cheek raising and lip corner pulling, to identify when a smile occurs.
Multimodal Emotion Analysis
The research analyzed emotion from three sources: audio (valence, arousal, dominance), eye gaze (VAD values based on prior eye movement), and text transcripts. This multi-faceted approach allowed researchers to see how smiling relates to different emotional signals across various data types.
Narrative Dynamics
Smiling was found to influence the structure of stories told by survivors. It reduced sentences about orientation and the strict past while increasing sentences related to evaluation and present reflection, showing how smiles shape the recounting of traumatic events.
Negative-Affect Smiles
Smiles associated with negative emotions (like sadness or fear) were linked to more positive emotional outcomes later on. This indicates that even smiles in difficult memories can be part of a process where positive feelings are eventually expressed.

Terminology

Summary

Smiling plays a critical role in regulating emotion and social interaction during traumatic recollection The study investigates when, where, and why 978 Holocaust survivors smile in video testimonies. p. 1

Where the paper says it

Taken together, we assert that smiling plays a critical role in regulating emotion and social interaction during traumatic recollection. 2 Dataset The video data of this study were curated from the USC Shoah Foundation’s Visual History Archive (VHA)1 , which contains over 55,000 video testimonies of survivors and witnesses of the Holocaust and other genocides recorded between primarily 1994 and 1999 in over 50 countries and in over 30 languages.

Study Objectives

The research aimed to create an automatic smile detection model from facial features with an F1 of 85% and annotate detected smiles under two established taxonomies of smiling. The study also produced narrative features on 1,083,417 transcript sentences as well as emotional valence from three different modalities: audio, eye gaze, and text transcript. p. 0

Where the paper says it

We create an automatic smile detection model from facial features with an F1 of 85% and annotate detected smiles under two established taxonomies of smiling. We produce narrative features on 1,083,417 transcript sentences as well as emotional valence from three different modalities: audio, eye gaze, and text transcript.

Smiling rates were associated with specific semantic topics, narrative structures, and temporal syntaxes across the corpus. p. 0
Where the paper says it

We produce narrative features on 1,083,417 transcript sentences as well as emotional valence from three different modalities: audio, eye gaze, and text transcript. Smiling rates are associated with specific semantic topics, narrative structures, and temporal syntaxes across the corpus.

The authors found that negative-affect smiles are associated with more positive subsequent valence trajectories across all three modalities. p. 0
Where the paper says it

Smiling rates are associated with specific semantic topics, narrative structures, and temporal syntaxes across the corpus. Smiles often occur during periods of intense negative affect and we find negative-affect smiles are associated with more positive subsequent valence trajectories across all three modalities.

Data Collection and Annotation

The video data were curated from the USC Shoah Foundation’s Visual History Archive (VHA) which contains over 55,000 video testimonies of survivors and witnesses of the Holocaust and other genocides recorded between primarily 1994 and 1999 in over 50 countries and in over 30 languages. The analysis focused on a subset of 978 English-language interviews with each interview split into an average 4. p. 1

Where the paper says it

Interview contents. We selected a subset of 978 English-language interviews with each interview split into an average 4.

1 tapes of about 30 minutes, totaling 1,965 hours of playtime. p. 1
Where the paper says it

We selected a subset of 978 English-language interviews with each interview split into an average 4. 1 tapes of about 30 minutes, totaling 1,965 hours of playtime.

Additional multimodal data included Facial Action Units (AUs) from OpenFace [5], Emotion from audio: Valence, Arousal, and Dominance (VAD) [44] emotion values, temporally-aligned [21, 37], and Emotion from eyegaze: temporally aligned values of VAD from five seconds of prior eye movement [42]. p. 1
Where the paper says it

• Original Audio: 44. 1 kHz stereo (AAC) [63] • Annotated Transcript: time-aligned human annotation and diarization [63] • Metadata keywords: minute-level annotations from a closed set of Holocaust-specific keywords [64] • Facial Action Units: 17 facial Action Units (AUs) [20] from OpenFace [5], captured at 30 fps • Emotion from audio: Valence, Arousal, and Dominance (VAD) [44] emotion values, temporally-aligned [21, 37] • Emotion from eyegaze: temporally aligned values of VAD from five seconds of prior eye movement [42] • Localized eye movement: subject eye movement that is normalized relative to inferred interviewer position [42] 3 Related Work and Pilot Study The modern scientific study of smiling began with the discovery of the “Duchenne smile” in 1862 [15], which contrasted smiles that engaged just the corners of the lip with those that also had raised cheeks and eyelid lowering.

Feature Engineering and Modeling

The researchers trained 108 lightweight classifiers to predict the three labels over a large sweep of features and models, including three architectures (MLP, GRU, CNN), three modality combinations (include/exclude audio emotion, eyegaze emotion, LLM-derived semantic features), and six temporal windows surrounding the smile with leave-oneout cross validation. The most predictive features for binary smile detection were the per-subject z-scores of AU06 (cheek raising) and AU12 (lip corner pulling), both alone achieving AUCs of 0. p. 3

Where the paper says it

Univariate experiments reveal that the most predictive features were the per-subject z-scores of AU06 (cheek raising, associated with the Duchenne (i. , genuine) smile [15, 16]) and AU12 (lip corner pulling), both alone achieving AUCs of 0.

809 and 0.756, respectively. p. 3
Where the paper says it

, genuine) smile [15, 16]) and AU12 (lip corner pulling), both alone achieving AUCs of 0. 756, respectively.

The best model for final smile detection used AUs + Audio, achieving an F1 of 0. p. 4
Where the paper says it

Final smile detection. We run the best model (AUs + Audio) on the entire 3,997 video set using the F1-maximizing 𝜃=0.

85.

Narrative Dynamics and Regulation

Smiling significantly reduces the rates of Orientation narrative sentences (-4.70%) and sentences set in the Strict Past (-6. p. 6

Where the paper says it

We observe two related patterns: first, smiling significantly reduces the rates of Orientation (-4. 70%) narrative sentences and sentences set in the Strict Past (-6.

15%), while it increases the rates of Evaluation narrative sentences (+5. p. 6
Where the paper says it

70%) narrative sentences and sentences set in the Strict Past (-6. 15%), and significantly increases the rates of Evaluation narrative sentences (+5.

62%) and sentences in the Present Reflection (+4. p. 5
Where the paper says it

5 Δ rate during smiles (%) *** *** *** *** (a) Narrative structure Orientation Complicating Action Evaluation Resolution Other −5 0 5 Δ rate during smiles (%) *** *** *** *** ** (b) Temporal syntax Strict Past Habitual Past Present Narration Present Reflection Present Reliving Figure 6: Change in rate of (a) narrative structure and (b) temporal syntax categories during smile vs. non-smile sentences.

84%). Smiling is associated with differences in eye dynamics and blink rates that vary with narrative valence. p. 1
Where the paper says it

valence trajectories of surrounding sentences both in narratives and present-day, and across the three modalities of audio, eye gaze, and transcript. Smiling is also associated with differences in eye dynamics and blink rates that vary with narrative valence.

Negative-affect smiles accompany more positive valence trajectories across all modalities and both emotion types. p. 5
Where the paper says it

2, we find which narrative structures and temporal syntaxes most occur with smiling; in §5. 3 we observe how negative-affect smiles improve valence trajectories across all modalities and both emotion types; and in §5.

Conclusion on Regulation

The study concludes that smiling plays a critical role in regulating emotion and social interaction during traumatic recollection. This regulation is demonstrated through the association of smiling with specific semantic topics, narrative structures, and temporal syntaxes across the corpus. p. 0

Where the paper says it

We produce narrative features on 1,083,417 transcript sentences as well as emotional valence from three different modalities: audio, eye gaze, and text transcript. Smiling rates are associated with specific semantic topics, narrative structures, and temporal syntaxes across the corpus.

Smiling typically occurs in emotionally congruent settings, but almost a fifth of all smiles occur during moments of negative valence. p. 0
Where the paper says it

We identify associations between smiling and two types of survivor emotion: the “narrative valence” of the historical content being recollected and the “present-day valence” of the in-the-moment social dynamics when interfacing with interviewers. We find smiling typically occurs in emotionally congruent settings, such as during topics associated with pre-war and childhood events, but that almost a fifth of all smiles occur during moments of negative valence.

The findings highlight how smiling acts as a mediator of emotion, both in the emotions embedded into recalled narratives and into the social dynamics captured between survivor and interviewer. p. 0
Where the paper says it

In this work, we investigate the social and narrative role that smiling plays during interviews of traumatic recollection with survivors of the Holocaust. We find this nonverbal behavior acts as a mediator of emotion, both in the emotions embedded into recalled narratives and into the social dynamics captured between survivor and interviewer.

Smiling is also associated with differences in eye dynamics and blink rates that vary with narrative valence. p. 1
Where the paper says it

valence trajectories of surrounding sentences both in narratives and present-day, and across the three modalities of audio, eye gaze, and transcript. Smiling is also associated with differences in eye dynamics and blink rates that vary with narrative valence.

The authors assert that smiling plays a critical role in regulating emotion and social interaction during traumatic recollection. p. 1
Where the paper says it

Smiling is also associated with differences in eye dynamics and blink rates that vary with narrative valence. Taken together, we assert that smiling plays a critical role in regulating emotion and social interaction during traumatic recollection.

The work provides insights into how smiling is an incredibly complex and rich expression of human communication. p. 7
Where the paper says it

8 Conclusion We study the role smiling plays for regulating the emotional and social dynamics of narrative recollection and discover the multimodal interactions of smiling with valence, physical markers, and narrative content. These findings underpin how smiling is an incredibly complex and rich expression of human communication and hope this work inspires future research on human smile.

Smiling is both an instinctual and learned behavior, influenced by cultural backgrounds and personal context. p. 7
Where the paper says it

Negative-valence smiling may function as a form of emotional regulation [19], which has long been linked to increased cognitive effort [50, 51]. 6 Cultural Differences and Limitations Smiling is both an instinctual and learned behavior [17, 23].

The analysis carries significant caution and ethical weight given the nature of the atrocity studied. p. 7
Where the paper says it

7 Safe and Responsible Innovation Statement In this work, we study Holocaust survivor video testimonies through facial, voice, and narrative processing. Analysis of this atrocity carries significant caution and ethical weight.

Smiling is a carefully constructed display of social expression that balances internal affect and the emotions one wishes to display to the world. p. 0
Where the paper says it

While we humans instinctually smile when we are happy [23], we also smile during periods of great distress, as reactions to surprise, and to mask our underlying emotions. Smiling, therefore, is not simply a valence barometer; it is a carefully constructed display of social expression that balances internal affect and the emotions one wishes to display to the world.

The work inspires future research on human smile. p. 7
Where the paper says it

8 Conclusion We study the role smiling plays for regulating the emotional and social dynamics of narrative recollection and discover the multimodal interactions of smiling with valence, physical markers, and narrative content. These findings underpin how smiling is an incredibly complex and rich expression of human communication and hope this work inspires future research on human smile.

Smiling, emotion, social interaction, traumatic recollection, multimodal interaction are keywords for this study. p. 0
Where the paper says it

Taken together, we conclude that smiling plays a critical role in regulating emotion and social interaction during traumatic recollection. CCS Concepts • Applied computing→Psychology; Sociology;• Human-centered computing; • Computing methodologies → Discourse, dialogue and pragmatics; Keywords smiling, emotion, social interaction, traumatic recollection, multimodal interaction ACM Reference Format: Marcus Ma, Emily Zhou, Leonard Ludwig, Julia Hörath, Christina Winkler, Kleanthis Avramidis, Tiantian Feng, Gabor Toth, Alina Bothe, and Shrikanth Narayanan.

The ACM Reference Format is Marcus Ma, Emily Zhou, Leonard Ludwig, Julia Hörath, Christina Winkler, Kleanthis Avramidis, Tiantian Feng, Gabor Toth, Alina Bothe and Shrikanth Narayanan. p. 0
Where the paper says it

Taken together, we conclude that smiling plays a critical role in regulating emotion and social interaction during traumatic recollection. CCS Concepts • Applied computing→Psychology; Sociology;• Human-centered computing; • Computing methodologies → Discourse, dialogue and pragmatics; Keywords smiling, emotion, social interaction, traumatic recollection, multimodal interaction ACM Reference Format: Marcus Ma, Emily Zhou, Leonard Ludwig, Julia Hörath, Christina Winkler, Kleanthis Avramidis, Tiantian Feng, Gabor Toth, Alina Bothe, and Shrikanth Narayanan.

2026. Smiling Regulates Emotion During Traumatic Recollection. p. 0
Where the paper says it

Smiling Regulates Emotion During Traumatic Recollection Marcus Ma mjma@usc. edu University of Southern California Los Angeles, California, USA Emily Zhou University of Southern California Los Angeles, California, USA Leonard Ludwig Freie Universität Berlin Berlin, Germany Julia Hörath Freie Universität Berlin Berlin, Germany Christina Winkler Freie Universität Berlin Berlin, Germany Kleanthis Avramidis University of Southern California Los Angeles, California, USA Tiantian Feng University of Southern California Los Angeles, California, USA Gabor Toth University of Luxembourg Esch-sur-Alzette, Luxembourg Alina Bothe Freie Universität Berlin Berlin, Germany Shrikanth Narayanan University of Southern California Los Angeles, California, USA Abstract We study when, where, and why 978 Holocaust survivors smile in video testimonies.

The paper is licensed under a Creative Commons Attribution 4. p. 0
Where the paper says it

Smiling Regulates Emotion During Traumatic Recollection. In INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (ICMI This work is licensed under a Creative Commons Attribution 4.

0 International License. p. 0
Where the paper says it

In INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (ICMI This work is licensed under a Creative Commons Attribution 4. 0 International License.

The ACM ISBN is 979-8-4007-2318-6/2026/10. p. 0
Where the paper says it

ICMI ’26, Napoli, Italy © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2318-6/2026/10 https://doi. p. 0

Where the paper says it

ICMI ’26, Napoli, Italy © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2318-6/2026/10 https://doi.

https://doi.org/10.1145/3776574. p. 0
Where the paper says it

ACM ISBN 979-8-4007-2318-6/2026/10 https://doi. 1145/3776574.

3831180’26), October 05–09, 2026, Napoli, Italy. p. 0
Where the paper says it

1145/3776574. 3831180 ’26), October 05–09, 2026, Napoli, Italy.

The work is supported by the USC Shoah Foundation, the Luxembourg National Research Fund and the German Research Foundation (Grant C24/ID/18896236/VOICES). p. 7
Where the paper says it

These findings underpin how smiling is an incredibly complex and rich expression of human communication and hope this work inspires future research on human smile. Acknowledgments This work was supported by the USC Shoah Foundation, the Luxembourg National Research Fund, the German Research Foundation (Grant C24/ID/18896236/VOICES), and the Deutsche Forschungsgemeinschaft (551119792).

The acknowledgments section notes that they used generative AI to assist with coding, references, and building the annotation tool. p. 7
Where the paper says it

We must consider the bias of cultural influence on smiling as well. We used generative AI to assist with coding, references, and building the annotation tool.

The paper is basic research understanding the role smiles play in traumatic narratives. p. 7
Where the paper says it

We do not disclose any of the survivors’ names or personally identifiable information but we anonymously acknowledge their contribution and honor the memory of those who did not survive. This work is basic research understanding the role smiles play in traumatic narratives, and as a result, we report only corpus-level and anonymous statistics.

The authors anonymously acknowledge their contribution and honor the memory of those who did not survive. p. 7
Where the paper says it

All video was provided by the USC Shoah Foundation’s Visual History Archive (VHA) under institutional agreement, with no new data collection. We do not disclose any of the survivors’ names or personally identifiable information but we anonymously acknowledge their contribution and honor the memory of those who did not survive.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Multimodal Smile Detection Model Enhancement: The automatic smile detection model can be improved by incorporating features from all three modalities—audio, eye gaze, and text transcript—to achieve higher accuracy than relying on facial features alone. This improved system can detect smiles with an F1 of 85% by leveraging emotion derived from audio which was found to consistently improve AUC in the final model comparison.

  2. Narrative Feature Integration for Contextual Understanding: The AI system can be enhanced to derive narrative features like Narrative era, Temporal syntax, and Oral structural narrative for every sentence, using LLM inference on transcripts. This allows the system to map smile rates against specific linguistic structures, such as observing that smile rate peaks during interview beginnings, reaches its nadir in the middle of the interview, which coincides with the highest proportion of wartime content.

  3. Valence Prediction Modality Selection: The system can be improved by dynamically selecting modalities for valence labeling based on their predictive power. Specifically, it should use LLM transcript outputs to predict narrative valence (53% accuracy) and audio emotion to predict present-day valence (72% accuracy), as these modalities were found to best align with the two emotion types.

  4. Smile Regulation Analysis: The improved system can analyze how smiling regulates emotion by mapping negative-affect smiles onto surrounding sentences and contrasting their valence trajectories. This allows the system to determine that valence trajectories are higher during smile clusters under both narrative and present-day conditions, suggesting smiling plays a critical role in regulating emotion and social interaction during traumatic recollection.

  5. Temporal Syntax Impact Assessment: The system can analyze how smiling affects temporal syntax by observing changes in sentence structure. This capability can identify that smiling significantly reduces the rates of Orientation narrative sentences and sentences set in the Strict Past, while simultaneously increasing the rates of Evaluation narrative sentences and sentences in the Present Reflection.

Abstract

We study when, where, and why 978 Holocaust survivors smile in video testimonies. We create an automatic smile detection model from facial features with an F1 of 85% and annotate detected smiles under two established taxonomies of smiling. We produce narrative features on 1,083,417 transcript sentences as well as emotional valence from three different modalities: audio, eye gaze, and text transcript. Smiling rates are associated with specific semantic topics, narrative structures, and temporal syntaxes across the corpus. Smiles often occur during periods of intense negative affect and we find negative-affect smiles are associated with more positive subsequent valence trajectories across all three modalities. Smiling reduces eye dynamics and blink rates, with both of these effects modulated by narrative valence. Taken together, we conclude that smiling plays a critical role in regulating emotion and social interaction during traumatic recollection.

Sources

Related papers