Smiling Regulates Emotion During Traumatic Recollection
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Smiling Regulates Emotion During Traumatic Recollection".
Jane: Here's the gist of the paper. Smiling plays a critical role in regulating emotion and social interaction during traumatic recollection The study investigates when, where, and why 978 Holocaust survivors smile in video testimonies.
Tom: Study Objectives The research aimed to create an automatic smile detection model from facial features with an F1 of 85% and annotate detected smiles under two established taxonomies of smiling.
Jane: The study also produced narrative features on 1,083,417 transcript sentences as well as emotional valence from three different modalities: audio, eye gaze, and text transcript.
Tom: Smiling rates were associated with specific semantic topics, narrative structures, and temporal syntaxes across the corpus.
Paper summary: Tom: So we're looking at this paper today, "Smiling Regulates Emotion During Traumatic Recollection," and it’s about how smiling actually plays a role in managing feelings and social interactions when people are remembering traumatic events.
Jane: That sounds really important because trauma is such a heavy thing to process, right? So what is the main idea here? What are they claiming about smiles in these videos of survivors?
Lu: The paper sets out to create an automatic model that can detect smiles from facial features with an F1 score of eighty-five percent, and then they label those detected smiles using two different ways to categorize them: one based on social function, and another based on how authentic the feeling behind the smile is.
Meng: Okay, so they are building a detection model first, which sounds like standard machine learning stuff. But what's the big payoff for us as people listening? What do these smiles actually tell us?
Lalam: The study produces narrative features from over one million sentences and emotional valence data from three different ways: audio, eye gaze, and text transcript. And they found that negative-affect smiles are linked to more positive feeling trajectories across all three of those ways.
Tom: Negative-affect smiles leading to more positive feelings—that’s a specific link they’re pointing out there. So when we see someone smile during a memory of something bad, it might actually be signaling something different than you'd think.
Jane: Exactly. It suggests that smiling isn't just random; it seems to be part of the way people regulate emotion when they are recalling those difficult times, which is what the title is about.
Lu: They looked at how smiles relate to specific topics in the story, like narrative structures and temporal syntaxes—which is just a fancy way of saying how sentences are put together over time. They found that smiling significantly reduces the rate of orientation sentences and sentences set in the strict past by forty-seven percent and sixty-one percent respectively.
Meng: Forty-seven percent reduction in those types of sentences—that’s a big chunk of narrative structure changing just because someone is smiling while recalling something traumatic. That’s what I need to know for deployment; how does that translate into a usable signal?
Lalam: The paper also showed that smiling is associated with differences in eye dynamics and blink rates, and these changes vary depending on the narrative valence, which means whether the story part they are looking at is positive or negative.
Paper summary: Tom: So we have smiles linked to specific narrative structures—like reducing sentences about where things are or when they happened—and also linked to how eyes move during those moments. That sounds like a complex feedback loop between what’s being said and how the person looks while saying it.
Jane: It really emphasizes that smiling acts as a mediator, bridging the internal emotion with the social interaction happening between the survivor and whoever is recording them, which is why they call it regulation during recollection.
Lu: The authors conclude that smiling plays a critical role in regulating emotion and social interaction during traumatic recollection because it’s associated with specific semantic topics, narrative structures, and temporal syntaxes across the corpus.
Meng: So we have these findings about how smiles change the way narratives are structured and how they affect emotional feeling, but what does this mean for someone just listening to a story without knowing all that technical jargon?
Jane: It means that when you’re hearing a difficult memory, the way someone chooses to smile—whether it's polite or something more authentic—tells you a lot about what they are actually feeling underneath the surface.
Tom: And we have to remember the context here: almost a fifth of all smiles in their data occurred during moments of negative valence, which means even in trauma, smiling isn't just a simple happy signal.
Lalam: The paper also notes that this smiling is often a carefully constructed display of social expression that tries to balance internal affect with the emotions one wants to show the world.
Lu: That idea—that it’s a constructed display—is interesting because it brings in the idea of learned behavior, influenced by cultural backgrounds and personal context, which is something we need to factor into any model we build for this.
Meng: If we were trying to build a system to analyze social dynamics during stress, knowing that the smile itself has a social function, like masking negative emotion or being polite for agreement, that’s crucial information for deployment.
Jane: It gives us a framework for understanding not just *what* people are saying, but *how* they are interacting with their emotions in real time during something really hard.
Tom: The paper also mentions the authors used generative AI to help with coding and building that annotation tool, which is a cool detail about how modern researchers use tools to handle this kind of complex data.
Paper summary: Lu: The study’s limitation, which the authors themselves flagged, is that they are limited to analyzing only English-speaking subjects who survived internment at various concentration camps. So we can’t generalize those specific findings easily across all trauma experiences globally right now.
Meng: That makes sense for a specific dataset, but if we want to make a model that works on real-world data from different countries, how much do we have to change about the feature engineering based on these results?
Lalam: The most predictive features they found for just detecting a smile were the per-subject z-scores of facial action units AU06, which is cheek raising, and AU12, which is lip corner pulling. Those two alone achieved AUCs of zero point eight zero nine and zero point seven five six respectively.
Tom: So those specific muscle movements are the strongest indicators for a binary smile detection before you even look at the whole narrative context. That gives us a concrete starting point for our model training, I like that specificity.
Jane: It grounds the abstract concept of "emotion regulation" into tangible facial muscle movements that we can actually measure and process with an AI model.
Lu: And they used a combination of Action Units plus audio emotion for their best final smile detection model, which achieved an F1 score of zero point eight five. That tells us that combining visual cues with vocal tone gives them the most accurate result on this specific task.
Meng: If we deploy that on a real video feed, running a model that combines facial action units and audio signals sounds doable, but what’s the latency like? How fast does it have to process those inputs for this to be useful in a live setting?
Tom: The paper didn't give us the exact latency numbers, but they focused on training models with different architectures like MLP, GRU, and CNN over six temporal windows around the smile. That suggests they were testing how deep the model needed to look into the video stream.
Jane: So while we know *what* features are important for detection—AU06 and AU12—the real complexity lies in how those features interact with the surrounding conversation and narrative structure, which is what Lu’s work on representation learning is about.
Lu: Precisely, the paper shows that smiles aren't just isolated events; they are woven into specific patterns of sentences and temporal syntax across different modalities like audio, eye gaze, and text transcript. That requires a model that can handle all three at once.
Paper summary: Meng: For production, running three separate emotion detectors—one for audio valence, one for eye gaze VAD values from the previous five seconds, and one for text semantic features—that sounds like it could be computationally expensive to run continuously on low-power devices.
Lalam: The study's main limitation is that they focused their analysis on a subset of nine hundred seventy-eight English-speaking survivors, so we can’t assume this exact relationship holds true for every single traumatic recollection across every culture or language without further testing <ref:2604.19019#pg1>.
Tom: It’s a strong finding, though. It moves the conversation past just looking at the raw emotion and starts looking at the specific social and structural patterns that accompany it in these memories.
Jane: And what this means for people who only listen to these kinds of studies is that smiling isn't just a surface expression; it’s a complex mechanism that helps manage how we process deep, painful experiences like trauma.
Lu: It opens up avenues for research into how we can build representations of social interaction during stress that are sensitive to these subtle facial cues and narrative dynamics.
Meng: If I were building this on real traffic, I’d be worried about the noise—the background chatter—interfering with detecting those subtle eye gaze changes or the tiny muscle movements in the face.
Tom: So, to summarize for our listeners today: smiling during traumatic memory isn't just a mood indicator; it's linked to specific sentence structures and eye dynamics, and negative smiles actually precede more positive feelings across different ways we can measure emotion.
Jane: That’s the core of this paper on "Smiling Regulates Emotion During Traumatic Recollection," showing that these small expressions are actually part of a larger system for regulating how we handle difficult memories.
Lu: The implication is that future models need to look beyond just detecting a smile, but understanding how that specific smile fits into the entire narrative flow and the emotional context of what is being recalled.
Meng: And for those of us building things, it means we can start prioritizing features like AU06 and AU12 when designing our emotion detection pipelines if we want to capture that foundational signal reliably.
Lalam: We're really confident this finding holds up because they tied the smile detection to established taxonomies of both social function and emotional authenticity, giving us a solid way to interpret what those detected smiles mean in practice.
Conclusion: Tom: So, this paper is about how smiling actually helps control your emotions when you’re remembering something traumatic like the Holocaust or other genocides.
Jane: It looks at video testimonies and tries to figure out when and why people smile in those moments because it seems like a really complex thing to study.
Tom: Yeah, the authors set out to build a model that could automatically spot those smiles from facial features with eighty-five percent accuracy.
Jane: And they didn't just stop there; they looked at how smiling relates to different parts of the story, like what kind of sentences people use or how their eyes move.
Tom: They found that negative feelings accompanied by a smile actually lead to more positive feelings later on across all those different ways they measured emotion.
Jane: So the main idea is that smiling isn't just random; it’s actually part of a system where people regulate their emotions while recalling something terrible.
Tom: It suggests that these smiles are tied to specific topics in the story and how time moves through the narrative structure.
Jane: And what this means for us is that we can start looking at smiles not just as an emotion itself, but as a signal about how someone is managing their feelings in a very difficult situation.
Tom: It’s showing that smiling acts like a mediator between what's happening inside the person and how they interact with others during those painful memories.
Jane: That leads us to think about how we can use this kind of analysis to better understand social dynamics during intense stress or memory recall.
University of Southern California · Freie über University Berlin
cs.MM
Submitted: 2026-04-21
Updated: 2026-11-10
Comments: Received Best Paper Runner-Up Award at ICMI 2026
Journal ref: In Proceedings of the 28th ACM International Conference on Multimodal Interaction (ICMI '26), October 5-9, 2026, Napoli, Italy
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Importance score: 78/100
The gist: Smiling plays a critical role in regulating emotion and social interaction during traumatic recollection The study investigates when, where, and why 978 Holocaust survivors smile in video testimonies.
Key concepts
- Automatic Smile Detection Model
- The study created an automated system using facial features to detect smiles in videos. It achieved high accuracy (F1 of 85%) by analyzing specific facial movements, such as cheek raising and lip corner pulling, to identify when a smile occurs.
- Multimodal Emotion Analysis
- The research analyzed emotion from three sources: audio (valence, arousal, dominance), eye gaze (VAD values based on prior eye movement), and text transcripts. This multi-faceted approach allowed researchers to see how smiling relates to different emotional signals across various data types.
- Narrative Dynamics
- Smiling was found to influence the structure of stories told by survivors. It reduced sentences about orientation and the strict past while increasing sentences related to evaluation and present reflection, showing how smiles shape the recounting of traumatic events.
- Negative-Affect Smiles
- Smiles associated with negative emotions (like sadness or fear) were linked to more positive emotional outcomes later on. This indicates that even smiles in difficult memories can be part of a process where positive feelings are eventually expressed.
Terminology
Summary
Smiling plays a critical role in regulating emotion and social interaction during traumatic recollection The study investigates when, where, and why 978 Holocaust survivors smile in video testimonies. p. 1 Taken
together, we assert that smiling plays a critical role in regulating
emotion and social interaction during traumatic recollection. 2 Dataset
The video data of this study were curated from the USC Shoah
Foundation’s Visual History Archive (VHA)1
, which contains over
55,000 video testimonies of survivors and witnesses of the Holocaust
and other genocides recorded between primarily 1994 and 1999 in
over 50 countries and in over 30 languages.Where the paper says it
Study Objectives
The research aimed to create an automatic smile detection model from facial features with an F1 of 85% and annotate detected smiles under two established taxonomies of smiling. The study also produced narrative features on 1,083,417 transcript sentences as well as emotional valence from three different modalities: audio, eye gaze, and text transcript. p. 0 We create an automatic smile detection model
from facial features with an F1 of 85% and annotate detected smiles
under two established taxonomies of smiling. We produce narrative features on 1,083,417 transcript sentences as well as emotional
valence from three different modalities: audio, eye gaze, and text
transcript. We produce narrative features on 1,083,417 transcript sentences as well as emotional
valence from three different modalities: audio, eye gaze, and text
transcript. Smiling rates are associated with specific semantic topics, narrative structures, and temporal syntaxes across the corpus. Smiling rates are associated with specific semantic topics, narrative structures, and temporal syntaxes across the corpus. Smiles often occur during periods of intense negative affect and
we find negative-affect smiles are associated with more positive
subsequent valence trajectories across all three modalities.Where the paper says it
Where the paper says it
Where the paper says it
Data Collection and Annotation
The video data were curated from the USC Shoah Foundation’s Visual History Archive (VHA) which contains over 55,000 video testimonies of survivors and witnesses of the Holocaust and other genocides recorded between primarily 1994 and 1999 in over 50 countries and in over 30 languages. The analysis focused on a subset of 978 English-language interviews with each interview split into an average 4. p. 1 Interview contents. We selected a subset of 978 English-language
interviews with each interview split into an average 4. We selected a subset of 978 English-language
interviews with each interview split into an average 4. 1 tapes of
about 30 minutes, totaling 1,965 hours of playtime. • Original Audio: 44. 1 kHz stereo (AAC) [63]
• Annotated Transcript: time-aligned human annotation
and diarization [63]
• Metadata keywords: minute-level annotations from a closed
set of Holocaust-specific keywords [64]
• Facial Action Units: 17 facial Action Units (AUs) [20] from
OpenFace [5], captured at 30 fps
• Emotion from audio: Valence, Arousal, and Dominance
(VAD) [44] emotion values, temporally-aligned [21, 37]
• Emotion from eyegaze: temporally aligned values of VAD
from five seconds of prior eye movement [42]
• Localized eye movement: subject eye movement that is
normalized relative to inferred interviewer position [42]
3 Related Work and Pilot Study
The modern scientific study of smiling began with the discovery
of the “Duchenne smile” in 1862 [15], which contrasted smiles that
engaged just the corners of the lip with those that also had raised
cheeks and eyelid lowering.Where the paper says it
Where the paper says it
Where the paper says it
Feature Engineering and Modeling
The researchers trained 108 lightweight classifiers to predict the three labels over a large sweep of features and models, including three architectures (MLP, GRU, CNN), three modality combinations (include/exclude audio emotion, eyegaze emotion, LLM-derived semantic features), and six temporal windows surrounding the smile with leave-oneout cross validation. The most predictive features for binary smile detection were the per-subject z-scores of AU06 (cheek raising) and AU12 (lip corner pulling), both alone achieving AUCs of 0. p. 3 Univariate experiments reveal that the most
predictive features were the per-subject z-scores of AU06 (cheek
raising, associated with the Duchenne (i. , genuine) smile [15,
16]) and AU12 (lip corner pulling), both alone achieving AUCs of
0. , genuine) smile [15,
16]) and AU12 (lip corner pulling), both alone achieving AUCs of
0. 756, respectively. Final smile detection. We run the best model (AUs + Audio) on the
entire 3,997 video set using the F1-maximizing 𝜃=0.Where the paper says it
Where the paper says it
Where the paper says it
Narrative Dynamics and Regulation
Smiling significantly reduces the rates of Orientation narrative sentences (-4.70%) and sentences set in the Strict Past (-6. p. 6 We observe two related patterns: first,
smiling significantly reduces the rates of Orientation (-4. 70%) narrative sentences and sentences set in the Strict Past (-6. 70%) narrative sentences and sentences set in the Strict Past (-6. 15%), and
significantly increases the rates of Evaluation narrative sentences
(+5. 5
Δ rate during smiles (%)
***
***
***
***
(a) Narrative structure
Orientation
Complicating
Action
Evaluation
Resolution
Other
−5
0
5
Δ rate during smiles (%)
***
***
***
***
**
(b) Temporal syntax
Strict
Past
Habitual
Past
Present
Narration
Present
Reflection
Present
Reliving
Figure 6: Change in rate of (a) narrative structure and (b) temporal syntax categories during smile vs. non-smile sentences. valence trajectories of surrounding sentences both in narratives
and present-day, and across the three modalities of audio, eye gaze,
and transcript. Smiling is also associated with differences in eye
dynamics and blink rates that vary with narrative valence. 2,
we find which narrative structures and temporal syntaxes most
occur with smiling; in §5. 3 we observe how negative-affect smiles
improve valence trajectories across all modalities and both emotion
types; and in §5.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion on Regulation
The study concludes that smiling plays a critical role in regulating emotion and social interaction during traumatic recollection. This regulation is demonstrated through the association of smiling with specific semantic topics, narrative structures, and temporal syntaxes across the corpus. p. 0 We produce narrative features on 1,083,417 transcript sentences as well as emotional
valence from three different modalities: audio, eye gaze, and text
transcript. Smiling rates are associated with specific semantic topics, narrative structures, and temporal syntaxes across the corpus. We identify associations between smiling and two types of survivor emotion: the “narrative valence” of the historical content being recollected and the “present-day valence” of the in-the-moment
social dynamics when interfacing with interviewers. We find smiling typically occurs in emotionally congruent settings, such as
during topics associated with pre-war and childhood events, but
that almost a fifth of all smiles occur during moments of negative valence. In this work, we investigate the social and narrative role
that smiling plays during interviews of traumatic recollection with
survivors of the Holocaust. We find this nonverbal behavior acts as
a mediator of emotion, both in the emotions embedded into recalled
narratives and into the social dynamics captured between survivor
and interviewer. valence trajectories of surrounding sentences both in narratives
and present-day, and across the three modalities of audio, eye gaze,
and transcript. Smiling is also associated with differences in eye
dynamics and blink rates that vary with narrative valence. Smiling is also associated with differences in eye
dynamics and blink rates that vary with narrative valence. Taken
together, we assert that smiling plays a critical role in regulating
emotion and social interaction during traumatic recollection. 8 Conclusion
We study the role smiling plays for regulating the emotional and
social dynamics of narrative recollection and discover the multimodal interactions of smiling with valence, physical markers, and
narrative content. These findings underpin how smiling is an incredibly complex and rich expression of human communication
and hope this work inspires future research on human smile. Negative-valence
smiling may function as a form of emotional regulation [19], which
has long been linked to increased cognitive effort [50, 51]. 6 Cultural Differences and Limitations
Smiling is both an instinctual and learned behavior [17, 23]. 7 Safe and Responsible Innovation Statement
In this work, we study Holocaust survivor video testimonies through
facial, voice, and narrative processing. Analysis of this atrocity carries significant caution and ethical weight. While we humans instinctually smile when we are happy [23], we also smile during periods of
great distress, as reactions to surprise, and to mask our underlying
emotions. Smiling, therefore, is not simply a valence barometer;
it is a carefully constructed display of social expression that balances internal affect and the emotions one wishes to display to the
world. 8 Conclusion
We study the role smiling plays for regulating the emotional and
social dynamics of narrative recollection and discover the multimodal interactions of smiling with valence, physical markers, and
narrative content. These findings underpin how smiling is an incredibly complex and rich expression of human communication
and hope this work inspires future research on human smile. Taken together, we conclude that
smiling plays a critical role in regulating emotion and social interaction during traumatic recollection. CCS Concepts
• Applied computing→Psychology; Sociology;• Human-centered
computing; • Computing methodologies → Discourse, dialogue and pragmatics;
Keywords
smiling, emotion, social interaction, traumatic recollection, multimodal interaction
ACM Reference Format:
Marcus Ma, Emily Zhou, Leonard Ludwig, Julia Hörath, Christina Winkler,
Kleanthis Avramidis, Tiantian Feng, Gabor Toth, Alina Bothe, and Shrikanth
Narayanan. Taken together, we conclude that
smiling plays a critical role in regulating emotion and social interaction during traumatic recollection. CCS Concepts
• Applied computing→Psychology; Sociology;• Human-centered
computing; • Computing methodologies → Discourse, dialogue and pragmatics;
Keywords
smiling, emotion, social interaction, traumatic recollection, multimodal interaction
ACM Reference Format:
Marcus Ma, Emily Zhou, Leonard Ludwig, Julia Hörath, Christina Winkler,
Kleanthis Avramidis, Tiantian Feng, Gabor Toth, Alina Bothe, and Shrikanth
Narayanan. Smiling Regulates Emotion During Traumatic Recollection
Marcus Ma
mjma@usc. edu
University of Southern California
Los Angeles, California, USA
Emily Zhou
University of Southern California
Los Angeles, California, USA
Leonard Ludwig
Freie Universität Berlin
Berlin, Germany
Julia Hörath
Freie Universität Berlin
Berlin, Germany
Christina Winkler
Freie Universität Berlin
Berlin, Germany
Kleanthis Avramidis
University of Southern California
Los Angeles, California, USA
Tiantian Feng
University of Southern California
Los Angeles, California, USA
Gabor Toth
University of Luxembourg
Esch-sur-Alzette, Luxembourg
Alina Bothe
Freie Universität Berlin
Berlin, Germany
Shrikanth Narayanan
University of Southern California
Los Angeles, California, USA
Abstract
We study when, where, and why 978 Holocaust survivors smile in
video testimonies. Smiling Regulates Emotion During Traumatic Recollection. In INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (ICMI
This work is licensed under a Creative Commons Attribution 4. In INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (ICMI
This work is licensed under a Creative Commons Attribution 4. 0 International License. ICMI ’26, Napoli, Italy
© 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2318-6/2026/10
https://doi. p. 0 ICMI ’26, Napoli, Italy
© 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2318-6/2026/10
https://doi. ACM ISBN 979-8-4007-2318-6/2026/10
https://doi. 1145/3776574. 1145/3776574. 3831180
’26), October 05–09, 2026, Napoli, Italy. These findings underpin how smiling is an incredibly complex and rich expression of human communication
and hope this work inspires future research on human smile. Acknowledgments
This work was supported by the USC Shoah Foundation, the Luxembourg National Research Fund, the German Research Foundation
(Grant C24/ID/18896236/VOICES), and the Deutsche Forschungsgemeinschaft (551119792). We must consider the bias of cultural influence
on smiling as well. We used generative AI to assist with coding,
references, and building the annotation tool. We do
not disclose any of the survivors’ names or personally identifiable
information but we anonymously acknowledge their contribution
and honor the memory of those who did not survive. This work
is basic research understanding the role smiles play in traumatic
narratives, and as a result, we report only corpus-level and anonymous statistics. All video was provided
by the USC Shoah Foundation’s Visual History Archive (VHA) under institutional agreement, with no new data collection. We do
not disclose any of the survivors’ names or personally identifiable
information but we anonymously acknowledge their contribution
and honor the memory of those who did not survive.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Multimodal Smile Detection Model Enhancement: The automatic smile detection model can be improved by incorporating features from all three modalities—audio, eye gaze, and text transcript—to achieve higher accuracy than relying on facial features alone. This improved system can detect smiles with an F1 of 85% by leveraging
emotion derived from audio
which was found to consistently improve AUC in the final model comparison. -
Narrative Feature Integration for Contextual Understanding: The AI system can be enhanced to derive narrative features like
Narrative era,
Temporal syntax,
andOral structural narrative
for every sentence, using LLM inference on transcripts. This allows the system to map smile rates against specific linguistic structures, such as observing thatsmile rate peaks during interview beginnings, reaches its nadir in the middle of the interview, which coincides with the highest proportion of wartime content.
-
Valence Prediction Modality Selection: The system can be improved by dynamically selecting modalities for valence labeling based on their predictive power. Specifically, it should use LLM transcript outputs to predict narrative valence (53% accuracy) and audio emotion to predict present-day valence (72% accuracy), as these modalities were found to
best align with the two emotion types.
-
Smile Regulation Analysis: The improved system can analyze how smiling regulates emotion by mapping negative-affect smiles onto surrounding sentences and contrasting their valence trajectories. This allows the system to determine that
valence trajectories are higher during smile clusters
under both narrative and present-day conditions, suggesting smilingplays a critical role in regulating emotion and social interaction during traumatic recollection.
-
Temporal Syntax Impact Assessment: The system can analyze how smiling affects temporal syntax by observing changes in sentence structure. This capability can identify that
smiling significantly reduces the rates of Orientation narrative sentences and sentences set in the Strict Past,
while simultaneously increasing the rates ofEvaluation narrative sentences and sentences in the Present Reflection.
Abstract
We study when, where, and why 978 Holocaust survivors smile in video testimonies. We create an automatic smile detection model from facial features with an F1 of 85% and annotate detected smiles under two established taxonomies of smiling. We produce narrative features on 1,083,417 transcript sentences as well as emotional valence from three different modalities: audio, eye gaze, and text transcript. Smiling rates are associated with specific semantic topics, narrative structures, and temporal syntaxes across the corpus. Smiles often occur during periods of intense negative affect and we find negative-affect smiles are associated with more positive subsequent valence trajectories across all three modalities. Smiling reduces eye dynamics and blink rates, with both of these effects modulated by narrative valence. Taken together, we conclude that smiling plays a critical role in regulating emotion and social interaction during traumatic recollection.
Sources
- Large Language Models for Oral History Understanding with Text Classification and Sentiment Analysis
- Vox-Profile: A Speech Foundation Model Benchmark for Characterizing Diverse Speaker and Speech Traits
- Encoding Emotion Through Self-Supervised Eye Movement Reconstruction
- gpt-oss-120b & gpt-oss-20b Model Card
- Exploring aspects of similarity between spoken personal narratives by disentangling them into narrative clause types
- Computational Analysis of Character Development in Holocaust Testimonies
Related papers
- When does the head leave? Shared timing and a movement clock in 360-degree video viewing
- Augmented Hybrid TV Services. Habits, Preferences, Opinions and Expectations of Spanish Consumers
- A Survey of Generative Categories and Techniques in Multimodal Generative Models
- ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling
- EyeNavGS: A 6-DoF Navigation Dataset and Record-n-Replay Software for Real-World 3DGS Scenes in VR
- Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis