ManzaiSet: A Multimodal Dataset of Viewer Responses to Japanese Manzai Comedy

arXiv:2510.18014 · cs.CV, cs.MM · Submitted 2025-10-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "ManzaiSet: A Multimodal Dataset of Viewer Responses to Japanese Manzai Comedy".

Tom: The gist ManzaiSet, the first large-scale multimodal dataset of viewer responses to Japanese manzai comedy,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So we’ve established the dataset is about Japanese manzai comedy, but who actually did this research? We're looking at ManzaiSet: A Multimodal Dataset of Viewer Responses to Japanese Manzai Comedy Kazuki Kawamura, Kengo Nakai, and Jun Rekimoto <ref:2510.18014#pg1>.

Jane: They’re researchers from Sony CSL Kyoto and The University of Tokyo, so they bring a solid academic background to this multimodal study <ref:2510.18014#pg2>.

Lu: What’s interesting about the authors is that they are tackling the cultural specificity directly in their research design <ref:2510.18014#pg3>.

Meng: From an engineering view, having researchers from different institutions working together on a large-scale multimodal dataset suggests a solid pipeline for handling complex data like this <ref:2510.18014#pg5>.

Lalam: This collaboration shows how different cultural perspectives can contribute to building datasets that are more comprehensive and less biased than what we usually have access to <ref:2510.18014#pg6>.

The paper's summary: Tom: Now let’s look at what they actually found in ManzaiSet. They used k-means clustering on the data and found three distinct viewer types: "High and Stable Appreciators" making up seventy-two point eight percent of the group <ref:2510.18014#pg6>.

Jane: That grouping is significant because they also showed these groups had a really big difference in how much their responses varied from one viewing session to the next, with a Brown–Forsythe p value less than zero point zero zero one <ref:2510.18014#pg6>.

Lu: The key finding there is that the "High and Stable Appreciators" had consistently high ratings, meaning minimal within-person variation in their appreciation style <ref:2510.18014#pg6>.

Meng: So it’s not just about who likes comedy versus who doesn't; it’s about stable appreciation styles, which is useful for understanding personalization algorithms <ref:2510.18014#pg4>.

Lalam: Plus, they found that even though there are these different viewer types, the automated humor classification—which involved seventy-seven instances and one hundred thirty-one labels—didn't show any statistically reliable differences between those types <ref:2510.18014#pg8>.

The paper's improvements: Tom: Moving on to what the authors suggest we should do with this information, they focus on improving how we handle cross-cultural emotion recognition <ref:2510.18014#pg2>.

Jane: They argue that because Japanese manzai shows cultural norms, any AI trying to understand emotion across different cultures can’t just train on one region’s data and expect it to work everywhere <ref:2510.18014#pg3>.

Lu: The implication for cross-cultural emotion recognition is big because this dataset provides a non-Western distribution against which models trained on Western data can be calibrated and stress-tested <ref:2510.18014#pg2>.

Meng: I see the practical part here as using those viewer types to condition personalized recommendation systems, making sure the content matches that stable appreciation style we found <ref:2510.18014#pg4>.

Lalam: And for me, it shows how this approach can help us build culture-aware emotion AI by moving beyond just Western data and focusing on these specific cultural patterns instead <ref:2510.18014#pg6>.

Conclusion: Tom: So to wrap up, the ManzaiSet paper gives us this large multimodal dataset of viewer responses to Japanese manzai comedy, showing three distinct viewer types and a positive viewing-order effect <ref:2510.18014#pg6>.

Jane: And the cultural context is huge here. It shows that for any AI trying to understand emotion across different cultures, you can’t just train on one region’s data and expect it to work everywhere <ref:2510.18014#pg3>.

Lu: The implication for cross-cultural emotion recognition is that this dataset lets us calibrate models trained on Western data against a non-Western distribution, which is really a necessary step for generalization <ref:2510.18014#pg2>.

Meng: I see the practical part in using those viewer types to condition personalized recommendation systems, making sure the content matches that stable appreciation style we found <ref:2510.18014#pg4>.

Lalam: And for me, it shows how this approach can help us build culture-aware emotion AI by focusing on these specific cultural patterns instead of just broad Western assumptions <ref:2510.18014#pg6>.

Tom: It really puts the focus back onto tailoring these systems to specific cultural contexts, not just chasing some global average <ref:2510.18014#pg6>.

Jane: Exactly. We’re looking at how engagement itself can drive satisfaction in a way that isn't always obvious from simple metrics like just watching time <ref:2510.18014#pg5>.

Lu: The positive momentum effect is a mechanism we need to consider when designing interactive entertainment experiences, not just passive viewing systems <ref:2510.18014#pg3>.

Meng: So the next step for us is figuring out how to integrate these viewer profiles into real-time content sequencing algorithms <ref:2510.18014#pg5>.

Lalam: That seems like where the real cultural impact lies, moving toward AI that understands nuance and stability in diverse human interactions <ref:2510.18014#pg6>.

Sony CSL Kyoto · The University of Tokyo · Yoshimoto Kogyo Holdings Co., Ltd.

cs.CV, cs.MM

Submitted: 2025-10-20

Updated: 2026-10-07

Comments: 7 pages, 3 figures. Revised description of dataset construction, exploratory analyses, and limitations. Presented at the ICCV 2025 Workshop on Affective & Behavior Analysis in-the-Wild (ABAW)

Journal ref: 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 42-50, 2025

DOI: 10.1109/ICCVW69036.2025.00010

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 79/100

The gist: The gist ManzaiSet, the first large-scale multimodal dataset of viewer responses to Japanese manzai comedy, captures facial videos and audio from 241 participants watching up to 10 professional

Key concepts

ManzaiSet
The first large-scale multimodal dataset of viewer responses to Japanese manzai comedy. It includes facial videos (640x360 resolution) and audio recordings from participants watching up to ten performances, designed to test affective computing outside Western contexts.
Viewer Typology
K-means clustering identified three distinct viewer types: 'High & Stable Appreciators' (72.8%), 'Low & Variable Decliners' (13.2%), and 'Variable Improvers' (14.0%). These groups show different patterns of rating behavior, suggesting viewer appreciation styles are stable traits rather than temporary states.
Positive Viewing-Order Effect
Analysis showed that the order in which viewers watched the performances positively impacted their ratings. The mean slope was 0.488 rating points per position, meaning viewing subsequent videos tended to increase satisfaction rather than causing fatigue.

Terminology

Summary

The gist

ManzaiSet, the first large-scale multimodal dataset of viewer responses to Japanese manzai comedy, captures facial videos and audio from 241 participants watching up to 10 professional performances in randomized order and addresses affective computing’s critical Western-centric bias.

Dataset Composition and Scale

The dataset comprises 2,301 viewing sessions from 241 participants with facial video recordings, as not all participants completed all 10 videos due to time constraints <ref:2510.18014#pg5>. The data collection involved capturing synchronized facial expressions (captured at 640 × 360 resolution, 25 fps) and audio (48 kHz sampling rate) during viewing sessions <ref:2510.18014#pg10>. Participants were recruited from a range of ages, and the final sample for analysis comprised 228 individuals who viewed at least 8 videos <ref:2510.18014#pg2>. The total recording duration amounted to 191.8 hours across the entire study <ref:2510.18014#pg5>.

Viewer Typology Identification

k-means clustering identified three distinct viewer types based on rating behaviors: “High & Stable Appreciators” (72.8%, n=166), “Low & Variable Decliners” (13.2%, n=30), and “Variable Improvers” (14.0%, n=32) <ref:2510.18014#pg6>. These groups exhibited significant heterogeneity of variance across within-person variability measures, as shown by the Brown–Forsythe test with a p. The “High & Stable Appreciators” were characterized by Consistently high ratings (mean ≈ 87.0, 95% CI: [86.0–88.1]) with minimal within-person variation <ref:2510.18014#pg6>.

Temporal Dynamics of Response

Individual-level analysis revealed a positive viewing-order effect, contradicting fatigue hypotheses <ref:2510.18014#pg3>. The mean slope was calculated to be 0.488 rating points per position (t(227) = 5.42, p. This positive momentum effect was confirmed by a permutation test (p. The integrated temporal analysis identified four distinct response patterns, including an ascending pattern with a mean slope of 0.8% <ref:2510.18014#pg3>.

Humor Mechanism Analysis

Analysis C investigated the relationships between specific humor mechanisms employed in manzai and the intensity of viewer responses <ref:2510.18014#pg4>. The analysis involved classifying 77 humor instances into nine categories, including unexpected twists (igai-sei, 35 instances) and repetition (tendon, 28 instances) <ref:2510.18014#pg10>. Despite the detailed modeling using a marginal logistic model with participant clustering and video fixed effects, no statistically reliable differences were detected between humor types after FDR correction (all adjusted p > 0.05) <ref:2510.18014#pg8>.

Implications for Affective Computing

The findings suggest that viewer-specific appreciation styles are trait-like rather than state-dependent, with a split-half reliability of r=0.866 <ref:2510.18014#pg6>. This stability recommends personalization as a first-class objective for recommendation systems <ref:2510.18014#pg4>. Furthermore, the observed momentum argues for engagement-aware sequencing in entertainment technology, suggesting that comedy viewing can create increasing satisfaction rather than fatigue <ref:2510.18014#pg5>. The dataset also provides a non-Western distribution against which to calibrate and stress-test models trained on Western data <ref:2510.18014#pg2>. This resource enables culturally-aware emotion AI development and personalized entertainment systems tailored to nonWestern contexts <ref:2510.18014#pg4>.

The paper presents ManzaiSet, the first large-scale multimodal dataset of viewer responses to Japanese manzai comedy, capturing facial videos and audio from 241 participants watching up to 10 professional performances in randomized order and addresses affective computing’s critical Western-centric bias. <ref:2510.18014#pg6>

How it works

The data collection was conducted remotely using a webbased system where participants recorded their facial expressions and audio synchronized with the comedy content <ref:2510.18014#pg10>. The experimental procedure required participants to access the application, enter credentials, and then record data while playing videos, providing subjective ratings after each video playback <ref:2510.18014#pg10>. The facial video data was processed using OpenFace 2.0 [2] to extract facial action units (AUs) and head pose parameters <ref:2510.18014#pg5>.

Key Findings Summary

  1. Three distinct viewer types were identified via k-means clustering, including “High & Stable Appreciators” (72.8%) <ref:2510.18014#pg6>.

  2. A positive viewing-order effect was found, with a mean slope of 0.488 rating points per position <ref:2510.18014#pg4>.

  3. No statistically reliable differences were detected between the nine humor categories after FDR correction <ref:2510.18014#pg8>.

This work represents a crucial step toward developing emotion AI that appreciates and adapts to cultural diversity. <ref:2510.18014#pg6>

The gist

ManzaiSet, the first large-scale multimodal dataset of viewer responses to Japanese manzai comedy, captures facial videos and audio from 241 participants watching up to 10 professional performances in randomized order and addresses affective computing’s critical Western-centric bias. <ref:2510.18014#pg6>

Key Findings Summary

  1. Three distinct viewer types were identified via k-means clustering, including “High & Stable Appreciators” (72.8%) <ref:2510.18014#pg6>.

  2. A positive viewing-order effect was found, with a mean slope of 0.

Improvements for AI systems

  1. Bold viewer-specific personalization based on trait-like profiles: Implement few-shot conditioning on a viewer’s recent signals (ratings, micro-expressions, vocal bursts) should lower error relative to global models by using the identified three viewer types ("High & Stable Appreciators," "Low & Variable Decliners, and Variable Improvers") as prior knowledge for personalized recommendation systems.

  2. Bold engagement-aware sequencing: Develop session controllers that anticipate rising satisfaction rather than fatigue, leveraging the finding that comedy viewing can create increasing satisfaction to optimize playlist ordering dynamically.

  3. Bold cross-cultural emotion recognition calibration: Calibrate and stress-test models trained on Western data by using ManzaiSet as a non-Western distribution, specifically focusing on domain adaptation and cross-domain evaluation protocols that quantify generalization gaps.

  4. Bold response-aligned performance training diagnostics: Create feedback systems for comedy rehearsal that summarize cues into per-instance curves timelocked to boke–tsukkomi cycles, laughter-onset rate/latency, and multimodal cues (e.g., AU dynamics, vocal bursts).

  5. Bold laugh-aware interactive agents: Design agents that can detect and anticipate laughables and schedule content to sustain momentum, modulating tone with light, culturally appropriate amusement displays based on the observed positive viewing-order effect.

Sources

Related papers