Build it, Break it, Repeat: Benchmarking and improving LLM-manipulated disinformation detection in social media posts

This paper adapts the Build it, Break it, Fix it framework into a Build it, Break it, Repeat (BiBiR) framework for evaluating and improving machine-generated disinformation detection in short-form social media posts. The framework consists of iterative sessions where builders develop detection models and breakers develop adversarial transformations to evade them, with five iterations completed in total.

Research Questions:

• RQ1: Which adversarial transformation techniques cause the largest reduction in detector performance?

• RQ2: Which adversarial techniques best preserve meaning while evading detection?

• RQ3: Between baseline transformer and contrastive model architectures, which is more effective?

• RQ4: What are the benefits of iterative BiBiR evaluation over static evaluation?

Datasets:

• bld data: Training data for detectors with 2,560 examples (1,280 human-written and 1,280 machine-generated) for baseline training, and 3,840 samples for contrastive learning models with DASS. Human data came from PHEME and Constraint datasets (1,296 posts total). Machine-generated counterparts were created using four LLMs (Grok-4.1, Llama-3.3-70B-Instruct, GPT-4o, DeepSeek-R1) with persona-engineered prompts.

• brk data: Evaluation set with 125 machine-generated original posts (MGO) and 125 human-generated original posts (HGO), expanded to 1.08M LLM-generated/altered posts across iterations.

Breakers' Techniques (four families):

1. Character-level perturbation (A3): Rule-based spacing edits including double spaces, spaces before punctuation, random newlines, and extra blank lines.

2. Lexical perturbation (B1-B3): Paraphrasing (B1 conservative, B3 aggressive) using LLaMA-3.1-8B-Instruct, and back-translation (B2) using OPUS-MT/NLLB models with intermediate languages including Arabic, Russian, Japanese, German, and Chinese.

3. Stylometric camouflage (C3): Emotion-based emoji injection using an emotion classifier (Emotion English DistilRoBERTa-base).

4. Prompt-based evasion (D1-D4): Tone-based rewriting (D1) and persona-based prompts of increasing refinement (D2 simple persona, D3 demographic-specific persona, D4 indirect persona). Personas included politician, college student, brain-rot 10-year old, and four-star military general.

Key Findings for RQ1 (Adversarial Techniques):

• Chained adversarial attacks were more effective than isolated transformations.

• The strongest breakers' configuration achieved 95.2% label flip rate (LFR) in iteration 5 (B3 D13 D43 B2 AR), reducing detector accuracy to 4.8%.

• Average detector accuracy decreased from 71.1% (iteration 3) to 60.2% (iteration 4) to 57.6% (iteration 5).

• Back-translation (B2) was the most reliable contributor to detector evasion, with Arabic back-translation (B2 AR) appearing in all max-LFR configurations.

• D4 (refined indirect persona) was the most effective persona-based technique family.

• B1 (original paraphrasing) showed no useful contribution (1% LFR), while B3 (revised paraphrasing) supported other techniques.

• A3 (whitespace editing) and C3 (emoji injection) had no positive ablation effect.

Key Findings for RQ2 (Semantic Preservation):

• Manual evaluation of 780 posts across iterations 3-5 using three criteria: ME1 (semantic preservation), ME2 (naturalness), ME3 (preservation of disinformation claim).

• Iteration 5 achieved the highest semantic preservation scores (ME1: 2.585, ME2: 3.912, ME3: 0.868).

• D4 standalone achieved ME1 of 2.855, ME2 of 3.990, and ME3 of 0.988, making it the best standalone technique family.

• Adding B2 to standalone techniques reduced semantic preservation: D4 B2 vs D4 showed ΔME1 of-0.108 and ΔME3 of-0.114.

• B2 ZH (Chinese back-translation) demonstrated the weakest semantic preservation and claim-retention results.

• 69 posts (8%) received ME3 score of 0 (claim not held), with 58% of these flipping labels—not successful evasion.

• E5-cosine similarity showed the strongest correlation with manual evaluation (Spearman's ρ = 0.489 for ME1, 0.453 for ME3).

• NLI contradiction labels (n=47, 6%) had average ME1 of 1.986 and ME3 of 0.620, compared to entailment (ME1 > 2.540, ME3 > 0.85).

Key Findings for RQ3 (Model Architectures):

• The Triplet(DASS) model was the best detector, achieving 72.68% accuracy and 84.18% F1 in iteration 5, outperforming the baseline by 15 percentage points.

• The baseline (e5-small-LoRA) dropped from 71.10% to 57.60% accuracy from iteration 3 to 5.

• Siamese(1:1) was the worst-performing model (24.94% accuracy in iteration 3, dropping to 16.99% in iteration 4).

• Siamese(TF-IDF) and Triplet(TF-IDF) showed moderate performance initially but degraded significantly (27.23% and 24.44% accuracy respectively by iteration 5).

• DASS was the only strategy maintaining accuracy above 70% across all iterations (81.52% → 72.94% → 72.75%).

• The key distinction was not contrastive architecture alone, but whether the model was trained to handle paraphrasing variations through DASS's dynamic anchor switching.

Key Findings for RQ4 (BiBiR vs Static Evaluation):

• Static benchmark evaluation significantly overestimated model robustness.

• The baseline classifier showed a 35% difference between builders' held-out data and iteration 5 dataset.

• Siamese(TF-IDF) showed a 68.5% difference, Triplet(TF-IDF) a 58% difference, and Triplet(DASS) a 15.6% difference.

• The iterative framework exposed weaknesses not observable under static conditions, including overfitting to discourse markers (mentions, hashtags, URLs) and imbalanced topics.

Vulnerabilities Discovered:

• Extreme personas like brain-rot 10-year old produced human-like, distorted outputs that tricked all models into flipping predictions.

• The college student and brain-rot personas exposed the biggest weaknesses by replicating human signals and informal styles.

• The politician persona maintained structure and formality that aligned closer to machine-generated posts.

Limitations:

• Scarcity of reliable human misinformation datasets (pre-LLM era data limited to PHEME, Constraint, TruthSeeker).

• Selection bias in brk data (only initially correctly classified examples retained).

• Small dataset sizes limited diversity and scale.

Future Work:

• Builders: Optimize Triplet(DASS) architecture, collect higher-quality human misinformation data.

• Breakers: Use semantic filtering pipeline for active rejection of posts, expand persona variety.

• BiBiR: Add validation stage for scalable semantic preservation checking using hybrid human/LLM approach.

Based on the paper, here are specific improvements to AI systems:

• Improvement: Train a detector using the Triplet(DASS) architecture with dynamic anchor switching, which maintains >70% accuracy across adversarial iterations.

• Capability: The system can detect machine-generated disinformation even when text is paraphrased, back-translated, or rewritten with personas, achieving 72.68% accuracy and 84.18% F1 against the strongest adversarial attacks.

• Improvement: Implement a chained attack pipeline combining refined indirect persona (D4) with conservative paraphrasing, using E5-cosine similarity as a filter to reject transformations with low semantic preservation (ρ = 0.489 correlation with human judgment).

• Capability: The system can generate adversarial examples that evade detection while maintaining high semantic preservation (ME1: 2.855, ME3: 0.988), enabling stress-testing of other NLP systems without destroying meaning.

• Improvement: Replace static benchmarks with a BiBiR-style iterative evaluation loop, running at least 5 adversarial cycles to expose overfitting to surface features (mentions, hashtags, URLs) that static tests miss.

• Capability: The system can identify robustness gaps of up to 68.5% between held-out and adversarially-tested performance, providing realistic deployment readiness scores for content moderation systems.

• Improvement: Add a persona-detection module that flags outputs matching extreme personas (e.g., brain-rot 10-year-old, demographic-specific styles) which currently trick all detectors into flipping predictions.

• Capability: The system can distinguish human-like distorted outputs from genuine human posts, reducing false positives in social media moderation by identifying persona-induced artifacts.

• Improvement: Train the detector to be robust against back-translation attacks, particularly Arabic (B2 AR), which was the most reliable evasion technique across all iterations.

• Capability: The system can maintain detection accuracy when content is translated through multiple languages (Arabic, Russian, Japanese, German, Chinese), preventing cross-lingual laundering of disinformation.

• Improvement: Integrate an NLI-based contradiction checker that rejects adversarial examples with ME3 scores below 0.85, since 58% of claim-breaking transformations are not successful evasions.

• Capability: The system can automatically filter out low-quality adversarial examples during training, improving data efficiency and preventing the detector from learning spurious patterns from semantically broken text.

• Improvement: Implement the DASS mechanism that switches between anchor examples during contrastive training, making the model learn paraphrase-invariant representations rather than surface-level features.

• Capability: The system can generalize to unseen paraphrasing styles, maintaining 72.75% accuracy even after 5 adversarial iterations, compared to 57.6% for static baseline models.

Detecting machine-generated disinformation on social media is increasingly difficult as large language models (LLMs) make it easier to generate and rewrite misleading content at scale. Static benchmark evaluations, measuring detector performance on fixed held-out datasets, do not capture how detectors behave when posts are deliberately transformed to evade classification. This paper adapts the Build it, Break it, Fix it framework into Build it, Break it, Repeat (BiBiR): iterative sessions designed to stress-test detectors' robustness under iterative adversarial conditions, evaluating whether models remain reliable when disinformation posts are systematically transformed to evade classification. Across five iterations, the findings show that the best adversarial breakers' transformations came from a combination of back-translation and LLM persona-based rewriting, with the best performing technique achieving a 95% label flip rate (LFR), whilst still preserving the meaning of the original posts. The best builders' model was a triplet contrastive model with a dynamic anchor switching (DASS) architecture, which achieved an average accuracy of 72.68%, outperforming the strong baseline (a fine-tuned e5-small-LoRA) by 15 percentage points on the most robust set of breakers' adversarial attacks. The results demonstrate that an iterative framework best exposes detector weaknesses and pushes robustness improvements; however, it may still require semantic preservation analysis to distinguish valid adversarial evasion from transformations that changed the original disinformation claims' meaning.