Subjective Multi-Bias Detection with Large Language Models

The paper introduces a project focused on detecting subjective bias in text, specifically identifying three types: framing bias, epistemological bias, and demographic bias. The authors use the WIKIBIAS dataset, which contains over 4,000 sentence pairs from Wikipedia edits, labeled by bias type (framing, epistemological, demographic, or no bias). The dataset is split into 5,028 training samples, 1,066 validation samples, and 2,104 test samples.

As a baseline, the authors fine-tune a BERT-base model (12 layers, 768 hidden dimensions, 12 self-attention heads, 110 million parameters) on the WIKIBIAS training set for three epochs. The baseline achieves a macro-F1 of 0.33 on the test set, revealing the task's challenges: comprehensive text understanding is required, and the dataset is highly imbalanced, with most errors coming from the minor classes (epistemological and demographic bias).

To improve performance, the authors develop an original approach called EnsembleLlama. They build three separate LLaMA2-MLP classifiers, each fine-tuned to detect one specific bias type. They freeze the pre-trained LLaMA2-7B model and add linear layers at the end, training only those linear layers. They design three system messages (prompts) that explain each bias type with an example. For instance, the prompt for framing bias states: "Framing bias is a type of subjective bias that uses of one-sided words or phrases containing a particular point of view. Here is an example: New York is the greatest state in the northeastern United States. Does the following sentence have framing bias?" Similar prompts are used for epistemological and demographic bias.

The authors initially tried training a single LLaMA-MLP to output three logits (one per bias type) and also tried outputting a label from 0 to 7 (combinations of biases), but both performed similarly to the baseline due to class imbalance. They then reformulated the output, training three separate classifiers (one per bias type) to address class imbalance, and stacked their results for the final detection. This stacking approach boosted performance significantly.

For fine-tuning, they used the HuggingFace meta-llama/Llama-2-7b-hf repository and FastChat, with batch size 2, max length 2048, initial learning rate 2e-5, and a cosine annealing scheduler. They ran experiments on two NVIDIA 80GB H100 GPUs, with three epochs taking about 1 hour.

The results show that EnsembleLlama achieves a macro-F1 of 0.59, which is 0.26 higher than the baseline (0.33). Class-level F1 scores also improved significantly: framing bias F1 improved from 0.49 to 0.75, epistemological bias from 0.25 to 0.52, and demographic bias from 0.25 to 0.49. The authors note that their model corrects many previous error cases, providing an example where the baseline misclassified an epistemological bias as framing bias, but EnsembleLlama correctly identified it as epistemological bias.

In the discussion, the authors analyze baseline errors: the baseline misclassified sentences as framing bias (e.g., an epistemological bias example) and missed multiple biases in a single sentence (e.g., a sentence with both framing and demographic biases). For EnsembleLlama, they note that while class imbalance is mitigated, the F-class F1 is still 1.4 times higher than E-class and D-class F1, indicating remaining imbalance between minor biases and no bias. They also acknowledge limitations: performance depends on LLaMA2's text understanding and knowledge, and simple stacking may not fully utilize mutual promotion between classifiers. They suggest future work on more advanced class imbalance methods, better ensemble learning, detecting bias at the single-word edit level, and extending to more datasets.

Based on this paper, here are the specific improvements I can make to an AI system, and what the improved system can do:

Improvements:

1. Decompose multi-label classification into per-class binary classifiers instead of a single multi-class output. This directly mitigates class imbalance by isolating rare classes (epistemological, demographic) and prevents the dominant no bias or framing classes from overwhelming the loss signal.

2. Use frozen large language models (LLaMA2-7B) with trainable linear heads rather than fine-tuning the entire model. This preserves the model's pre-trained linguistic knowledge while reducing overfitting on small, imbalanced datasets.

3. Inject task-specific definitions and examples via system prompts for each bias type before classification. This primes the model to attend to the precise linguistic cues (e.g., one-sided words for framing, epistemic modality for epistemological) rather than relying on generic text understanding.

4. Stack independent classifiers' outputs (one per bias type) to form a final multi-label prediction. This allows each classifier to specialize without interference from other bias types, and the final decision can be thresholded per class.

5. Apply cosine annealing learning rate scheduling with a low initial learning rate (2e-5) to stabilize training of the linear heads, preventing drastic updates that could disrupt the frozen backbone's representations.

What the improved AI system can do:

• Detect multiple subjective biases simultaneously in a single sentence (e.g., both framing and demographic bias) with high accuracy, instead of missing one when both are present.

• Achieve a macro-F1 of 0.59 (vs. 0.33 baseline), meaning it correctly identifies rare bias types (epistemological F1=0.52, demographic F1=0.49) that are typically missed by standard fine-tuned models.

• Provide explainable bias detection by leveraging the prompt definitions, so the system can justify why a sentence is biased (e.g., contains one-sided words like 'greatest' → framing bias).

• Operate efficiently—training only linear layers on frozen LLaMA2-7B takes 1 hour on 2 H100 GPUs, making it feasible for real-world deployment where computational resources are limited.

• Generalize to other subjective text domains (e.g., news articles, social media) because it relies on LLaMA2's broad language understanding rather than dataset-specific patterns, and the per-class binary setup can be extended to new bias types without retraining the entire system.

In this project, we delved into the pervasive challenge of bias detection within the text content. More specifically, our focus lies on the identification of subjective bias, a type of bias that introduces improper attitudes or portrays a statement at odds with the actual truth. The subjective bias can jeopardize the authenticity and reliability of texts, leading to misconceptions and potential social tensions, especially when expressed through offensive language. Following prior work [1], we tackled with three different types of subjective biases in text: (1) framing bias with the use of one-sided words or phrases containing a particular point of view; (2) epistemological bias which includes subtle linguistic features that can affect the believability of the texts; (3) demographic bias with word/phrase usage under presuppositions of a particular demographic factor (i.e., gender or religion). In terms of the data we utilize, the input consists of texts that may harbor subjective biases. The output is a classification or annotation that reveals the presence or absence of such biases within the provided content. More specifically, we detected three different types of multi-span biases in corpus WIKIBIAS [2] with more than 4,000 sentence pairs from Wikipedia edits. The data is labelled by bias type for span pairs with the following categories: (1) framing bias, (2) epistemological bias, (3) demographic bias, and (4) no bias. The project codes are released at https://github.com/HoningJade/LLM-Bias-Type-Classification.