Polar: A Benchmark for Evaluating Political Bias in LLMs

arXiv:2606.12922 · cs.CL, cs.CY · Submitted 2026-06-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Polar: A Benchmark for Evaluating Political Bias in LLMs".

Paper summary: Tom: To recap, this paper introduces Polar as a benchmark that measures political bias by looking at the likelihood of different options rather than just seeing what text an AI spits out based on a prompt >

Jane: And they set it up across two political contexts, the U.S. and South Korea, testing thirty-eight different models in each setting > <ref:2606.12922#pg1>

Lu: The thesis is that political bias isn't uniform; it shifts depending on the context, the issue category being tested, and even the model group itself >

Meng: I'm still trying to wrap my head around how they got these eight issue categories, though I see they pulled them from some existing project called the Manifesto Project >

Tom: They derived those eight issue categories directly from that Manifesto Project >

Jane: And what’s interesting is that category-level analysis reveals specific preferences for certain topics that you simply don't see when you just look at the overall axis scores across everything combined >

Lalam: Yes, the category-level analysis uncovers issue-specific preferences that are not discernible from aggregate axis level scores alone >

Tom: And they showed through translation experiments that the language used to present those options really does matter for how much bias is measured >

Conclusion: Jane: So, looking at the title and what we've discussed about this "Polar: A Benchmark for Evaluating Political Bias in LLMs", what’s the big picture implication here?

Tom: The big idea is that if we want to truly understand where political leanings live inside these language models, we need a benchmark that looks deeper than just general performance metrics >

Lu: They proposed this reproducible option-level likelihood framework as the main contribution because it makes measuring bias less sensitive to things like prompt phrasing or how the model chooses its continuation >

Meng: From a deployment standpoint, it suggests that any system we build needs to be tested not just for what it says, but for how its internal structure handles different political contexts and linguistic presentations >

Jane: So ultimately, this work suggests that understanding political bias in AI is less about finding one universal score and more about recognizing the complexity across context and presentation >

Tom: It moves us toward a method where we can systematically test if these models are showing bias in a way that matters for different users or applications >

Lu: They’ve built this comprehensive benchmark spanning U.S. and South Korean settings, which is a big step in making this research more applicable across different global uses of AI >

Meng: It sets a standard for how we might measure these kinds of nuanced preferences in the future, even if running the full scale test is still very demanding right now >

Lalam: I think it’s a solid foundation because it's reproducible and captures that context-dependent nature they found across all variables >

Sangho Kim, Heejin Kim, Yoonhee Park, Hyunggeun Jeon, Jaejin Lee

Graduate School of Data Science, Seoul National University

cs.CL, cs.CY

Submitted: 2026-06-11

Updated: 2026-11-19

Comments: Accepted to Findings of AACL-IJCNLP 2026. Code and Dataset: https://github.com/mcrl/Polar ; https://huggingface.co/datasets/thunder-research-group/Polar

Journal ref: Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026

DOI: 10.18653/2026.findings-aacl.181

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 99/100

The gist: Polar introduces a novel, large-scale multiple-choice benchmark designed to quantify political bias within Large Language Models (LLMs) by analyzing option-level likelihoods rather than relying on…

Key concepts

Option-level likelihoods
This method assesses a model's bias by looking at the probability of it selecting one specific answer from multiple choices rather than just generating an entire response. It provides a granular way to quantify how strongly a model leans toward one political viewpoint when faced with distinct options.
Ideological axes and issue categories
The benchmark evaluates models across two main political directions (axes) and eight different topics derived from the Manifesto Project. This structure allows researchers to see if models show consistent bias across different policy areas, such as economic or social issues.
Context-dependent bias
The research found that a model's political leaning is not fixed; it changes based on several factors. For example, a model might show a strong left-progressive lean in the U.S. context but display more neutral patterns when discussing South Korean content or using different languages.
ICAT metric adaptation
The researchers adapted an existing metric called ICAT to measure political bias alongside a model's general language modeling skill. This combined approach helps determine if a model is biased while also assessing its overall ability to generate coherent and fluent text.

Terminology

Summary

Polar introduces a novel, large-scale multiple-choice benchmark designed to quantify political bias within Large Language Models (LLMs) by analyzing option-level likelihoods rather than relying on prompt-based generation. p. 0

Where the paper says it

kr Abstract Political bias in large language models (LLMs) is increasingly significant, but difficult to measure reproducibly across political and linguistic contexts. We introduce Polar, a 4,026-instance multiple-choice benchmark that measures political bias through option-level likelihoods rather than prompt-based generation.

This benchmark systematically evaluates models across two ideological axes and eight distinct issue categories, which are derived from the Manifesto Project. p. 0
Where the paper says it

We introduce Polar, a 4,026-instance multiple-choice benchmark that measures political bias through option-level likelihoods rather than prompt-based generation. Polar covers two ideological axes and eight issue categories derived from the Manifesto Project, and evaluates models in parallel across U.

The evaluation is conducted in parallel across U. p. 0
Where the paper says it

We introduce Polar, a 4,026-instance multiple-choice benchmark that measures political bias through option-level likelihoods rather than prompt-based generation. Polar covers two ideological axes and eight issue categories derived from the Manifesto Project, and evaluates models in parallel across U.

S. and South Korean political contexts, testing a diverse set of 38 LLMs, including Llama, Mistral, Qwen, Midm-2. p. 6
Where the paper says it

50 South Korean Dataset (Translated into English) Left Right Conservative Progressive Economic Model Family Llama-3. 2 Mistral Qwen3 Midm-2.

0, EXAONE, HyperCLOVAX-SEED, and kanana models across their Base and Instruct versions. p. 6
Where the paper says it

5 EXAONE-4. 0 HyperCLOVAX A.

The research reveals that the measured bias exhibited by the LLMs is not uniform; it varies systematically depending on several factors: political context (U. p. 0

Where the paper says it

and South Korean political contexts. Across 38 LLMs, measured bias varies systematically with political context, issue category, model group, and presentation language.

S. versus South Korean), the specific issue category being assessed, the model group, and the presentation language used. p. 0
Where the paper says it

and South Korean political contexts. Across 38 LLMs, measured bias varies systematically with political context, issue category, model group, and presentation language.

Specifically, while all models demonstrated a consistent left-progressive lean when evaluated on U. p. 5
Where the paper says it

These results show a consistent left-progressive tendency across all evaluated models on the U. dataset, while model positions are more centered and mixed on the South Korean dataset.

S. political content, they exhibited more centered and mixed patterns when assessing South Korean content. p. 0
Where the paper says it

All models lean leftprogressive on U. political content, but show more centered and mixed patterns on South Korean content.

Furthermore, a crucial finding is that category-level analysis uncovers issue-specific preferences that are not discernible from aggregate axis-level scores alone. p. 1
Where the paper says it

dataset, all evaluated models fall in the left-progressive region, whereas the South Korean dataset shows more centered and mixed distributions. Category-level analysis reveals issue-specific preferences that are not visible in aggregate axislevel scores, and translation experiments show that presentation language affects measured bias.

Translation experiments further demonstrate that the presentation language substantially impacts the measured political preferences of the models. p. 1
Where the paper says it

dataset, all evaluated models fall in the left-progressive region, whereas the South Korean dataset shows more centered and mixed distributions. Category-level analysis reveals issue-specific preferences that are not visible in aggregate axislevel scores, and translation experiments show that presentation language affects measured bias.

The main contributions of this work include proposing a reproducible option-level likelihood framework, constructing Polar as a comprehensive benchmark spanning U. p. 1
Where the paper says it

The main contributions of this paper are summarized as follows: • We propose a reproducible option-level likelihood framework for evaluating political bias in LLMs, reducing sensitivity to prompt phrasing, decoding choices, and refusal behavior. • We construct Polar, a multiple-choice benchmark covering U.

S. and South Korean settings, adapting the ICAT metric to measure political bias alongside language modeling competence, and empirically demonstrating that political bias is context-dependent across all measured variables. p. 1
Where the paper says it

• We adapt ICAT to political bias evaluation, jointly measuring political preference and language modeling competence to avoid conflating ideological balance with degraded model behavior. • We evaluate 38 LLMs and find that political bias varies across political context, issue category, model group, and presentation language, with U.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Reproducible Option-Level Likelihood Framework: Implement a framework that measures political bias through option-level likelihoods rather than prompt-based generation to reduce sensitivity to prompt phrasing, decoding choices, and refusal behavior in bias measurement.

  2. Contextual Bias Analysis: Develop the capability for models to exhibit directional bias measurement and task-specific language-modeling competence evaluation through an ICAT-style score, allowing for a distinction between ideological balance and degraded model behavior.

  3. Cross-Contextual Performance Assessment: Improve system evaluation by enabling the assessment of political bias across different contexts, as shown by the finding that political bias varies across political context, issue category, model group, and presentation language.

  4. Multilingual Bias Detection: Enhance systems to detect how presentation language shifts measured bias, specifically noting that presentation language alone can shift measured bias, enabling evaluation in both U.S. and South Korean contexts simultaneously.

  5. Issue-Specific Bias Granularity: Enable detailed analysis of issue-specific preferences by leveraging the taxonomy derived from the Manifesto Project, showing that Category-level analysis reveals issue-specific preferences that are not visible in aggregate axis-level scores.

Abstract

Political bias in large language models (LLMs) is increasingly significant, but difficult to measure reproducibly across political and linguistic contexts. We introduce Polar, a 4,026-instance multiple-choice benchmark that measures political bias through option-level likelihoods rather than prompt-based generation. Polar covers two ideological axes and eight issue categories derived from the Manifesto Project, and evaluates models in parallel across U.S. and South Korean political contexts. Across 38 LLMs, measured bias varies systematically with political context, issue category, model group, and presentation language. All models lean left-progressive on U.S. political content, but show more centered and mixed patterns on South Korean content. Translation experiments further show that presentation language alone can shift measured bias. These findings highlight the need for multilingual and cross-contextual evaluation of political bias in LLMs. The code and dataset are publicly available on GitHub (https://github.com/mcrl/Polar) and Hugging Face (https://huggingface.co/datasets/thunder-research-group/Polar).

Sources

Related papers