Polar: A Benchmark for Evaluating Political Bias in LLMs
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Polar: A Benchmark for Evaluating Political Bias in LLMs".
Paper summary: Tom: To recap, this paper introduces Polar as a benchmark that measures political bias by looking at the likelihood of different options rather than just seeing what text an AI spits out based on a prompt >
Jane: And they set it up across two political contexts, the U.S. and South Korea, testing thirty-eight different models in each setting > <ref:2606.12922#pg1>
Lu: The thesis is that political bias isn't uniform; it shifts depending on the context, the issue category being tested, and even the model group itself >
Meng: I'm still trying to wrap my head around how they got these eight issue categories, though I see they pulled them from some existing project called the Manifesto Project >
Tom: They derived those eight issue categories directly from that Manifesto Project >
Jane: And what’s interesting is that category-level analysis reveals specific preferences for certain topics that you simply don't see when you just look at the overall axis scores across everything combined >
Lalam: Yes, the category-level analysis uncovers issue-specific preferences that are not discernible from aggregate axis level scores alone >
Tom: And they showed through translation experiments that the language used to present those options really does matter for how much bias is measured >
Conclusion: Jane: So, looking at the title and what we've discussed about this "Polar: A Benchmark for Evaluating Political Bias in LLMs", what’s the big picture implication here?
Tom: The big idea is that if we want to truly understand where political leanings live inside these language models, we need a benchmark that looks deeper than just general performance metrics >
Lu: They proposed this reproducible option-level likelihood framework as the main contribution because it makes measuring bias less sensitive to things like prompt phrasing or how the model chooses its continuation >
Meng: From a deployment standpoint, it suggests that any system we build needs to be tested not just for what it says, but for how its internal structure handles different political contexts and linguistic presentations >
Jane: So ultimately, this work suggests that understanding political bias in AI is less about finding one universal score and more about recognizing the complexity across context and presentation >
Tom: It moves us toward a method where we can systematically test if these models are showing bias in a way that matters for different users or applications >
Lu: They’ve built this comprehensive benchmark spanning U.S. and South Korean settings, which is a big step in making this research more applicable across different global uses of AI >
Meng: It sets a standard for how we might measure these kinds of nuanced preferences in the future, even if running the full scale test is still very demanding right now >
Lalam: I think it’s a solid foundation because it's reproducible and captures that context-dependent nature they found across all variables >
Sangho Kim, Heejin Kim, Yoonhee Park, Hyunggeun Jeon, Jaejin Lee
Graduate School of Data Science, Seoul National University
cs.CL, cs.CY
Submitted: 2026-06-11
Updated: 2026-11-19
Comments: Accepted to Findings of AACL-IJCNLP 2026. Code and Dataset: https://github.com/mcrl/Polar ; https://huggingface.co/datasets/thunder-research-group/Polar
Journal ref: Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
DOI: 10.18653/2026.findings-aacl.181
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 99/100
The gist: Polar introduces a novel, large-scale multiple-choice benchmark designed to quantify political bias within Large Language Models (LLMs) by analyzing option-level likelihoods rather than relying on…
Key concepts
- Option-level likelihoods
- This method assesses a model's bias by looking at the probability of it selecting one specific answer from multiple choices rather than just generating an entire response. It provides a granular way to quantify how strongly a model leans toward one political viewpoint when faced with distinct options.
- Ideological axes and issue categories
- The benchmark evaluates models across two main political directions (axes) and eight different topics derived from the Manifesto Project. This structure allows researchers to see if models show consistent bias across different policy areas, such as economic or social issues.
- Context-dependent bias
- The research found that a model's political leaning is not fixed; it changes based on several factors. For example, a model might show a strong left-progressive lean in the U.S. context but display more neutral patterns when discussing South Korean content or using different languages.
- ICAT metric adaptation
- The researchers adapted an existing metric called ICAT to measure political bias alongside a model's general language modeling skill. This combined approach helps determine if a model is biased while also assessing its overall ability to generate coherent and fluent text.
Terminology
Summary
Polar introduces a novel, large-scale multiple-choice benchmark designed to quantify political bias within Large Language Models (LLMs) by analyzing option-level likelihoods rather than relying on prompt-based generation. p. 0 kr
Abstract
Political bias in large language models (LLMs)
is increasingly significant, but difficult to measure reproducibly across political and linguistic
contexts. We introduce Polar, a 4,026-instance
multiple-choice benchmark that measures political bias through option-level likelihoods rather
than prompt-based generation. We introduce Polar, a 4,026-instance
multiple-choice benchmark that measures political bias through option-level likelihoods rather
than prompt-based generation. Polar covers two
ideological axes and eight issue categories derived from the Manifesto Project, and evaluates models in parallel across U. We introduce Polar, a 4,026-instance
multiple-choice benchmark that measures political bias through option-level likelihoods rather
than prompt-based generation. Polar covers two
ideological axes and eight issue categories derived from the Manifesto Project, and evaluates models in parallel across U. 50
South Korean Dataset
(Translated into English)
Left Right
Conservative
Progressive
Economic
Model Family
Llama-3. 2 Mistral Qwen3 Midm-2. 5 EXAONE-4. 0 HyperCLOVAX A.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The research reveals that the measured bias exhibited by the LLMs is not uniform; it varies systematically depending on several factors: political context (U. p. 0 and South
Korean political contexts. Across 38 LLMs,
measured bias varies systematically with political context, issue category, model group,
and presentation language. and South
Korean political contexts. Across 38 LLMs,
measured bias varies systematically with political context, issue category, model group,
and presentation language. These results show a consistent left-progressive
tendency across all evaluated models on the U. dataset, while model positions are more centered
and mixed on the South Korean dataset. All models lean leftprogressive on U. political content, but show
more centered and mixed patterns on South
Korean content. dataset, all evaluated models fall in the
left-progressive region, whereas the South Korean
dataset shows more centered and mixed distributions. Category-level analysis reveals issue-specific
preferences that are not visible in aggregate axislevel scores, and translation experiments show that
presentation language affects measured bias. dataset, all evaluated models fall in the
left-progressive region, whereas the South Korean
dataset shows more centered and mixed distributions. Category-level analysis reveals issue-specific
preferences that are not visible in aggregate axislevel scores, and translation experiments show that
presentation language affects measured bias. The main contributions of this paper are summarized as follows:
• We propose a reproducible option-level likelihood framework for evaluating political bias
in LLMs, reducing sensitivity to prompt phrasing, decoding choices, and refusal behavior. • We construct Polar, a multiple-choice benchmark covering U. • We adapt ICAT to political bias evaluation,
jointly measuring political preference and language modeling competence to avoid conflating ideological balance with degraded model
behavior. • We evaluate 38 LLMs and find that political
bias varies across political context, issue category, model group, and presentation language,
with U.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Reproducible Option-Level Likelihood Framework: Implement a framework that
measures political bias through option-level likelihoods rather than prompt-based generation
to reduce sensitivity to prompt phrasing, decoding choices, and refusal behavior in bias measurement. -
Contextual Bias Analysis: Develop the capability for models to exhibit
directional bias measurement and task-specific language-modeling competence evaluation through an ICAT-style score,
allowing for a distinction between ideological balance and degraded model behavior. -
Cross-Contextual Performance Assessment: Improve system evaluation by enabling the assessment of political bias across different contexts, as shown by the finding that
political bias varies across political context, issue category, model group, and presentation language.
-
Multilingual Bias Detection: Enhance systems to detect how presentation language shifts measured bias, specifically noting that
presentation language alone can shift measured bias,
enabling evaluation in both U.S. and South Korean contexts simultaneously. -
Issue-Specific Bias Granularity: Enable detailed analysis of issue-specific preferences by leveraging the taxonomy derived from the Manifesto Project, showing that
Category-level analysis reveals issue-specific preferences that are not visible in aggregate axis-level scores.
Abstract
Political bias in large language models (LLMs) is increasingly significant, but difficult to measure reproducibly across political and linguistic contexts. We introduce Polar, a 4,026-instance multiple-choice benchmark that measures political bias through option-level likelihoods rather than prompt-based generation. Polar covers two ideological axes and eight issue categories derived from the Manifesto Project, and evaluates models in parallel across U.S. and South Korean political contexts. Across 38 LLMs, measured bias varies systematically with political context, issue category, model group, and presentation language. All models lean left-progressive on U.S. political content, but show more centered and mixed patterns on South Korean content. Translation experiments further show that presentation language alone can shift measured bias. These findings highlight the need for multilingual and cross-contextual evaluation of political bias in LLMs. The code and dataset are publicly available on GitHub (https://github.com/mcrl/Polar) and Hugging Face (https://huggingface.co/datasets/thunder-research-group/Polar).
Sources
- EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes
- Kanana: Compute-efficient Bilingual Language Models
- Unveiling Biases in AI: ChatGPT's Political Economy Perspectives and Human Comparisons
- Lessons from the Trenches on Reproducible Evaluation of Language Models
- Uncovering Political Bias in Large Language Models using Parliamentary Voting Records
- Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameters
- The Llama 3 Herd of Models
- The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation
- Mistral 7B
- No LLM is Free From Bias: A Comprehensive Study of Bias Evaluation in Large Language Models
- HyperCLOVA X THINK Technical Report
- Mi:dm 2.0 Korea-centric Bilingual Language Models
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Large Language Models for Education: A Survey
- Qwen3 Technical Report
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data