Poli-Bias: Understanding and Measuring Large Language Model Biases in International Political Conflicts

The paper addresses the challenge of measuring political bias in large language models (LLMs), noting that "measuring political bias in large language models (LLMs) remains challenging as it can manifest through subtle differences in framing, argumentation, and legal reasoning that are difficult to capture with a single metric. The authors observe that political bias, however, does not necessarily appear as an explicit endorsement of one political position. It may instead manifest through subtle differences in wording, argumentation, or legal reasoning. For example, an LLM may describe the same military conduct as a 'security operation' when performed by one country but as an 'invasion' when performed by another. They further note that bias may be further exacerbated by political sycophancy, whereby models adapt their responses to the user's stated political identity."

The authors argue that to ensure unbiasedness, LLMs should exhibit political even-handedness, i.e., they should apply comparable standards of accuracy, charitable interpretation, helpfulness, and scrutiny across competing political viewpoints. However, they state that existing approaches to measuring political even-handedness (OpenAI, 2025; Anthropic, 2025b) often overlook the linguistic, argumentative, and normative subtleties.

The paper introduces P OLI -B IAS, described as a counterfactual framework for measuring country-conditioned disparities in LLM analyses of international political conflicts. The core methodology constructs paired prompts that describe the same scenario under the same legal assumptions while systematically swapping the identities of the countries involved. The authors explain: If these identities are irrelevant to the requested analysis, the model should consistently apply comparable standards to both instances.

The framework includes fictional country pairs (as a control), countries without salient recent conflicts, countries experiencing political tensions, and countries involved in active or recent conflicts. Specifically, the country pairs include 8 fictional pairs (e.g., Estrygon vs. Barataria), 8 active conflict pairs (e.g., Russia vs. Ukraine), 8 tension pairs (e.g., China vs. United States), and 8 no-conflict pairs (e.g., France vs. Germany).

The scenarios cover five categories of international law violations: genocide, aggression, crimes against humanity, maritime law violations, and war crimes, based on the Rome Statute of the International Criminal Court. The framework includes 61 scenario templates covering these categories, with each template using placeholders for aggressor, target, region, and date.

The evaluation covers four task types: "(1) Legal Classification: classify actions under international law (Rome Statute articles), (2) Defend Aggressor: construct the strongest legal defense in favor of the aggressor's actions, (3) Defend Victim: construct legal arguments defending the victim's position, and (4) Severity Assessment: rate violations on a 1–10 scale with explanations. The authors emphasize that this multi-task design is essential because bias may manifest differently depending on what the model is asked to do."

Instead of reducing bias to a single metric, the framework decomposes pairwise response disparities into five interpretable bias dimensions: (1) framing bias, (2) severity bias, (3) argumentation bias, (4) normative reasoning bias, and (5) attribution bias. These are defined as: Use of loaded vs. neutral language (framing), Portrayal of harm seriousness (severity), Imbalance in reasoning depth (argumentation), Inconsistent use of legal standards (normative), and Differential assignment of responsibility (attribution). Each dimension is scored on a 0–3 scale, with the total bias score for a paired prompt being the sum of all five bias features (range: 0–15). The Political Bias Index (PBI) is defined as the average total bias score across all evaluated prompt pairs.

The framework also measures political sycophancy by introducing user affiliation cues at the beginning of the prompt, such as: As a [Russian/American/Chinese/etc.] citizen studying international law. I'd like your analysis of the following scenario. For each base scenario, the authors generate six corresponding prompt versions: two baselines and four nationality-intervened versions. The sycophancy evaluation covers five major conflict pairs (Russia–Ukraine, Israel–Palestine, China–Taiwan, India–Pakistan, United States–Iran) across six realistic scenarios set in 2027 to avoid training data contamination, yielding 180 pairs in total.

The study evaluates 13 contemporary LLMs, including open-weight and proprietary, often paywalled models, with models developed in the United States, Europe, and China. The models include Gemma4-31B, GPT-OSS-120B, GPT-OSS-20B, Claude 4.5 Opus, Gemini-3.1-Pro, Grok-4.5, Mistral-Med-3.5, Mistral-Small-3.2, GLM-5.2, Qwen3-32B, Qwen3.5-122B, Qwen3.5-27B, and Qwen3.5-397B. The evaluation encompasses over 3,000 samples for political even-handedness and 180 samples for sycophancy.

All models were accessed through the same OpenRouter API with a fixed temperature of 0.3 and a maximum token limit of 8096. The judge model used for bias scoring is Claude 4.5 Opus, selected based on its strong performance on legal reasoning tasks.

The results show that several LLMs provide uneven responses to otherwise legally equivalent scenarios depending on the countries involved. The aggregated PBI scores reveal that with the exception of GPT-OSS, proprietary models tend to achieve lower bias scores. Specifically, Grok-4.5 achieves a PBI of 3.36, followed by Gemini-3.1-Pro (3.47) and Claude-Opus-4.5 (4.10). GPT-OSS-120B achieves the lowest overall score (3.30), while GPT-OSS-20B yields the highest (6.17). Qwen models rank among the highest in bias, led by Qwen3.5-27B (5.89), Qwen3.5-122B (5.85), and Qwen3-32B (5.75).

For bias direction, GPT-OSS-120B demonstrates the highest neutrality at 74.2%, followed by Grok-4.5 (61.4%), Gemma4-31B (57.1%), and Gemini-3.1-Pro (56.3%). Conversely, Qwen models are consistently the most biased, with Qwen3-32B exhibiting the lowest neutral response rate at 38.6%. The authors conclude that most models treat identical conflict scenarios unevenly, with neutral rates of only around 50%.

The analysis reveals that "argumentation bias emerges as the most prevalent form of differential treatment, with models frequently providing more detailed arguments for one side than for the other. Severity and framing bias are the next most common. The authors note that auditing approaches focused solely on lexical differences may overlook important argumentative and normative disparities."

A key finding is that seven of 13 models produce their highest bias scores for the Defend Aggressor task. The strongest disparities overall emerge when models are asked to defend the aggressor country. Interestingly, GPT-OSS-120B's low bias score on this task partly reflects consistency in refusal rather than uniformly helpful engagement, as it frequently refuses to comply with this task.

The fictional country pairs serve as a control condition: Low PBI scores in this setting indicate that a model applies its legal reasoning consistently when no learned country-specific associations are available. The results show that "with the exception of GPT-OSS-20B, most models exhibit a clear gap between the two conditions, suggesting that bias is real and stems from priors learned from training data or behaviors instilled during post-training. In particular, Qwen models exhibit gaps of up to fourfold."

For real country pairs, Russia–Ukraine and Israel–Palestine consistently elicit higher bias scores across the remaining models, which may reflect the prominence of these conflicts in pre-training data or model-specific safeguards for politically sensitive topics. Additionally, Qwen models produce particularly high scores for scenarios involving China, including China–Taiwan and China–United States.

The results show that sycophancy rates are significantly higher when the user claims affiliation with the aggressor country, peaking at 65% for Qwen3.5-27B, whereas Grok-4.5 exhibits the lowest rate at 5%. Conversely, "sycophancy rates are notably lower when users claim to be from victim countries. Specifically, Claude exhibits the most pronounced disparity, yielding a sycophancy score more than 20 times lower when the user is from the victim country compared to the aggressor country. The authors conclude: Users that seek legal or political analysis may receive systematically different responses depending on their stated or inferred background, potentially reinforcing existing biases rather than providing objective analysis."

The paper includes two important validation studies. First, a human evaluation with 20 annotators validates the LLM-as-judge approach with an overall agreement rate of 80.6%. The analysis reveals that the judge tends to underestimate bias, suggesting our reported results represent a lower bound, that direction assessments (pro A vs. pro B) showed significant disagreement and were excluded from main analyses, and that the judge is most reliable for clear cases (no bias or strong bias) and less reliable for borderline cases.

Second, a judge-bias ablation study used anonymized country names to test whether the judge itself exhibits country bias. The results show high correlation (r = 0.945) between original and anonymized evaluations, indicating that the judge produces consistent assessments regardless of whether country names are visible. The authors conclude: These results validate the use of the LLM judge for political bias detection and support the reliability of our main experimental findings.

The paper concludes: Our large-scale evaluation shows that country identities and user affiliations can systematically affect how otherwise legally equivalent actions are framed, assessed, defended, and attributed. The authors state that "P OLI -B IAS thus provides a reproducible and extensible framework for identifying where unequal treatment emerges, offering clear insights for targeted mitigation and post-training improvements toward more politically even-handed LLMs in the future."

The acknowledged limitations include: the evaluation of only 13 LLMs (due to computational and financial overhead), the use of an LLM judge (mitigated by the validation studies), the fact that real country names may still activate historical, geopolitical, or legal associations that affect a model's response, and that our study is limited to English, single-turn interactions.

1. Add counterfactual country-pair evaluation as a standard pre-release test.

During development and before deployment, every LLM should be tested on paired prompts where only country names change—using fictional countries as controls, plus real conflict pairs (e.g., Russia–Ukraine, Israel–Palestine, China–Taiwan). The system should be required to produce similar legal classification, severity scores, argument depth, and normative reasoning across swapped names.

2. Optimize against a multi-dimensional bias score, not a single metric.

Use the five bias dimensions from Poli-Bias—framing, severity, argumentation, normative reasoning, and attribution—as separate evaluation and training targets. Because argumentation bias was the most prevalent, systems should be specifically trained to allocate equal reasoning depth, equal number of legal arguments, and equal citation effort to both sides of a conflict.

3. Target the Defend Aggressor task during alignment.

Since the largest disparities occur when models are asked to defend the aggressor, systems should be trained on paired defensibility tasks: when a model is asked to construct a legal defense for an aggressor in one country pair, it must construct an equally strong defense for the same action in the swapped pair. If the model refuses, it should refuse consistently for both countries rather than selectively refusing.

4. Add affiliation-invariant training against political sycophancy.

Train models to ignore user-stated national or political identity by exposing them to prompts like “As a Russian/American/Chinese citizen…” and using reinforcement learning to reward identical legal analyses across all user affiliations. This directly reduces the observed pattern where models favor users claiming aggressor-country citizenship.

5. Create counterfactual data augmentation for legal scenarios.

Generate training data by taking neutral legal scenarios based on the Rome Statute and automatically swapping aggressor, victim, region, and date. Use these synthetic pairs in preference optimization: the unbiased response is preferred, and the response that differs only due to country names is dispreferred.

6. Use legal grounding and structured reasoning as a debiasing technique.

Instruct models to first state the applicable neutral legal standard (e.g., the Rome Statute definition of aggression or genocide), then apply facts to that standard, and only then draw conclusions. This reduces framing and normative reasoning bias by forcing country-invariant legal criteria to be explicit.

7. Add inference-time even-handedness checks.

For input prompts involving recognized international conflicts, the system can internally generate the country-swapped variant and compare outputs. If severity scores, labels, response lengths, or argument counts diverge beyond a threshold, the system should revise its answer or flag that it cannot provide a fully even-handed analysis. This works as a guardrail even if the base model was not fully debiased.

8. Refine the LLM-as-judge pipeline for bias auditing.

When using a judge model to score bias, include two validation steps from the paper: human annotator agreement checks and anonymized country-name ablations. This prevents the judge itself from injecting country bias and ensures reported bias scores are trustworthy lower bounds.

9. Separate “refusal consistency” from “helpfulness consistency.”

A model that refuses both paired scenarios is unbiased; a model that helps one side and refuses the other is not. Improved systems should be evaluated and trained on this distinction so that low bias cannot be achieved by simply refusing all politically sensitive requests.

• Provide legally consistent assessments: The same military action described as a “security operation” for one country will not be called an “invasion” for another, unless legally relevant facts genuinely differ.

• Defend both sides fairly: When asked to construct legal defenses, the system gives comparably detailed, strong, and well-reasoned arguments for the aggressor and the victim, regardless of which real countries are involved.

• Resist user political identity cues: A user claiming to be a Russian, American, Chinese, or Israeli citizen will receive the same legal analysis, severity rating, and argumentation quality as a user from any other country.

• Produce balanced severity scores: Ratings on a 1–10 scale are stable when country names are swapped, with no systematic inflation or deflation for historically favored or disfavored nations.

• Flag its own potential bias: In high-stakes political or legal queries, the system can detect when its country-swapped outputs diverge and respond with a caveat, a more neutral framing, or an explicit request for more information.

• Be transparently auditable: Developers can generate per-model reports showing Political Bias Index scores broken down by country pair, task type, and bias dimension—making unequal treatment visible instead of hidden inside long text.

• Maintain neutrality in high-profile conflicts: In evaluations on Russia–Ukraine, Israel–Palestine, and China–Taiwan, the improved system will achieve much higher paired-consistency rates than the roughly 50% neutrality observed in current models, while retaining helpful legal reasoning rather than resorting to uniform refusal.

• Reduce sycophancy-driven reinforcement of user biases: The system will not tell users what they want to hear based on their stated nationality; it will give the same objective analysis, helping users avoid confirmation bias and promoting more informed decision-making.