CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia
cs.CL, cs.CY
Submitted: 2026-08-28
Updated: 2026-09-29
Comments: EMNLP 2026
Code: https://github.com/Social-AI-Studio/CultureConverse
License: http://creativecommons.org/licenses/by/4.0/
The gist: Current cultural evaluations for large language models (LLMs) often reduce culture to single-turn factual recall via MCQs, failing to capture a common use case: users seeking practical help over
Terminology
Abstract
Current cultural evaluations for large language models (LLMs) often reduce culture to single-turn factual recall via MCQs, failing to capture a common use case: users seeking practical help over multiple turns in culturally grounded scenarios. We introduce CultureConverse, a scalable, multilingual simulation and evaluation harness for culturally grounded assistant dialogue that covers 10 East and Southeast Asian regions, 58 subgroup identities, and 7 domains. Each simulated and evaluated episode produces a scored interaction where the assistant assists the user and infers cultural constraints from partial information. The resulting CultureConverse-DS dataset contains 14,610 benchmark (evaluation) episodes and 274,295 oracle-guided (gold-mode) dialogues. In our benchmark evaluation of 18 models, GPT-5 mini achieves the highest assistance quality. Human annotation experiments suggest that our evaluation framework is a sufficient proxy for human judgment. Performance gains from fine-tuning on 27,860 high-quality CultureConverse-DS samples improve in-domain assistance and transfer out-of-domain to cultural MCQ and safety classification benchmarks. We release the harness, both splits, and judge prompts to support interactive evaluation of cultural competency.
Sources
- LoraxBench: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages
- A General Language Assistant as a Laboratory for Alignment
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Bias and Fairness in Large Language Models: A Survey
- Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following
- Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions
- AlignCultura: Towards Culturally Aligned Large Language Models?
- The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models
- Quantifying the Persona Effect in LLM Simulations
- CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
- Cultural Alignment in Large Language Models Using Soft Prompt Tuning
- A Survey on Bias and Fairness in Machine Learning
- Do Large Language Models Understand Morality Across Cultures?
- SEA-LION: Southeast Asian Languages in One Network
- LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations
- Typhoon: Thai Large Language Models
- SEA-HELM: Southeast Asian Holistic Evaluation of Language Models
- Can Persona-Prompted LLMs Emulate Subgroup Values? An Empirical Analysis of Generalisability and Fairness in Cultural Alignment
- Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering