Polar: A Benchmark for Evaluating Political Bias in LLMs
summary
The gist
Polar introduces a novel, large-scale multiple-choice benchmark designed to quantify political bias within Large Language Models (LLMs) by analyzing option-level likelihoods rather than relying on…
In short
Polar created a new benchmark to measure political bias in Large Language Models (LLMs) by analyzing how likely models are to choose specific options, not just generating text based on prompts. Testing 38 models across U.S. and South Korean contexts showed that bias varies significantly by political context, issue type, model group, and language used.
Key concepts
- Option-level likelihoods
- This method assesses a model's bias by looking at the probability of it selecting one specific answer from multiple choices rather than just generating an entire response. It provides a granular way to quantify how strongly a model leans toward one political viewpoint when faced with distinct options.
- Ideological axes and issue categories
- The benchmark evaluates models across two main political directions (axes) and eight different topics derived from the Manifesto Project. This structure allows researchers to see if models show consistent bias across different policy areas, such as economic or social issues.
- Context-dependent bias
- The research found that a model's political leaning is not fixed; it changes based on several factors. For example, a model might show a strong left-progressive lean in the U.S. context but display more neutral patterns when discussing South Korean content or using different languages.
- ICAT metric adaptation
- The researchers adapted an existing metric called ICAT to measure political bias alongside a model's general language modeling skill. This combined approach helps determine if a model is biased while also assessing its overall ability to generate coherent and fluent text.
Terminology used across episodes
This episode discusses
- Polar: A Benchmark for Evaluating Political Bias in LLMs · Paper Radio
- EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes
- Kanana: Compute-efficient Bilingual Language Models
- Unveiling Biases in AI: ChatGPT's Political Economy Perspectives and Human Comparisons
- Lessons from the Trenches on Reproducible Evaluation of Language Models
- Uncovering Political Bias in Large Language Models using Parliamentary Voting Records
- Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameters
- The Llama 3 Herd of Models · Paper Radio
- The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation
- Mistral 7B
- No LLM is Free From Bias: A Comprehensive Study of Bias Evaluation in Large Language Models
- HyperCLOVA X THINK Technical Report
- Mi:dm 2.0 Korea-centric Bilingual Language Models
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Large Language Models for Education: A Survey
- Qwen3 Technical Report
The paper
Polar: A Benchmark for Evaluating Political Bias in LLMs · Read on arXiv
Sangho Kim, Heejin Kim, Yoonhee Park, Hyunggeun Jeon, Jaejin Lee
Graduate School of Data Science, Seoul National University
Political bias in large language models (LLMs) is increasingly significant, but difficult to measure reproducibly across political and linguistic contexts. We introduce Polar, a 4,026-instance multiple-choice benchmark that measures political bias through option-level likelihoods rather than prompt-based generation. Polar covers two ideological axes and eight issue categories derived from the Manifesto Project, and evaluates models in parallel across U.S. and South Korean political contexts. Across 38 LLMs, measured bias varies systematically with political context, issue category, model group, and presentation language. All models lean left-progressive on U.S. political content, but show more centered and mixed patterns on South Korean content. Translation experiments further show that presentation language alone can shift measured bias. These findings highlight the need for multilingual and cross-contextual evaluation of political bias in LLMs. The code and dataset are publicly available on GitHub (https://github.com/mcrl/Polar) and Hugging Face (https://huggingface.co/datasets/thunder-research-group/Polar).
DOI: 10.18653/2026.findings-aacl.181
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Polar: A Benchmark for Evaluating Political Bias in LLMs".
Paper summary: Tom: To recap, this paper introduces Polar as a benchmark that measures political bias by looking at the likelihood of different options rather than just seeing what text an AI spits out based on a prompt >
Jane: And they set it up across two political contexts, the U.S. and South Korea, testing thirty-eight different models in each setting > <ref:2606.12922#pg1>
Lu: The thesis is that political bias isn't uniform; it shifts depending on the context, the issue category being tested, and even the model group itself >
Meng: I'm still trying to wrap my head around how they got these eight issue categories, though I see they pulled them from some existing project called the Manifesto Project >
Tom: They derived those eight issue categories directly from that Manifesto Project >
Jane: And what’s interesting is that category-level analysis reveals specific preferences for certain topics that you simply don't see when you just look at the overall axis scores across everything combined >
Lalam: Yes, the category-level analysis uncovers issue-specific preferences that are not discernible from aggregate axis level scores alone >
Tom: And they showed through translation experiments that the language used to present those options really does matter for how much bias is measured >
Conclusion: Jane: So, looking at the title and what we've discussed about this "Polar: A Benchmark for Evaluating Political Bias in LLMs", what’s the big picture implication here?
Tom: The big idea is that if we want to truly understand where political leanings live inside these language models, we need a benchmark that looks deeper than just general performance metrics >
Lu: They proposed this reproducible option-level likelihood framework as the main contribution because it makes measuring bias less sensitive to things like prompt phrasing or how the model chooses its continuation >
Meng: From a deployment standpoint, it suggests that any system we build needs to be tested not just for what it says, but for how its internal structure handles different political contexts and linguistic presentations >
Jane: So ultimately, this work suggests that understanding political bias in AI is less about finding one universal score and more about recognizing the complexity across context and presentation >
Tom: It moves us toward a method where we can systematically test if these models are showing bias in a way that matters for different users or applications >
Lu: They’ve built this comprehensive benchmark spanning U.S. and South Korean settings, which is a big step in making this research more applicable across different global uses of AI >
Meng: It sets a standard for how we might measure these kinds of nuanced preferences in the future, even if running the full scale test is still very demanding right now >
Lalam: I think it’s a solid foundation because it's reproducible and captures that context-dependent nature they found across all variables >
More episodes
- 2603.28113-The multiplicative Lipschitz bound as an inductive bias
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound