Build it, Break it, Repeat: Benchmarking and improving LLM-manipulated disinformation detection in social media posts

Authors: Kevin Thomas, Milosz Kasprzyk, Ruel C Igbokwe Onuigbo, Elliott Pert, Cameron Tovey, João A. Leite, Olesya Razuvayevskaya, Carolina Scarton

Venue: arXiv cs.SI (preprint), 2026 — arXiv:2608.09510

TL;DR

This paper presents BiBiR (Build-it-Break-it-Repeat), an iterative adversarial benchmarking framework for evaluating machine-generated text detectors under adversarial conditions relevant to social media. Two new datasets enable fine-tuning detectors and testing adversarial robustness. Results show that chained adversarial techniques achieve 95.2% label-flip rate, and iterative evaluation reveals more realistic detector vulnerabilities than static benchmarks.

Contributions

1. BiBiR framework — an iterative adversarial evaluation methodology where breakers develop attack techniques and builders develop defenses, repeated over five iterations to surface evolving vulnerabilities in detection models.

2. Two new datasets — (1) bld_data: 2,560 baseline examples for training detectors; 3,840 for contrastive learning with dynamic anchor switching (DASS); (2) brk_data: 1,080 machine-generated and altered posts from comprehensive adversarial generation for evaluation.

3. Adversarial attack taxonomy — four technique families with 14 variants: character-level perturbations (spacing, punctuation edits); lexical perturbations (paraphrasing, back-translation); stylometric camouflage (emoji injection); prompt-based evasion (tone/persona manipulation).

4. Robust detector architectures — siamese networks, triplet networks, and triplet networks with dynamic anchor switching (DASS) strategy that adaptively switches anchor selection to improve robustness against paraphrasing.

5. Semantic preservation analysis — automatic evaluation pipeline using NLI, token-length ratio, ES-cosine similarity, and BLERT to verify adversarial transformations preserve original meaning while evading detection.

6. Comparative evaluation — demonstrated that iterative breaker-builder rounds expose detector vulnerabilities missed by static benchmarks; chained attacks more effective than isolated techniques.

Method

The BiBiR framework organizes machine-generated text detection into five staged iterations. In each round, a breakers team develops adversarial transformation techniques, while a builders team designs increasingly robust detection models. The key innovation is this iterative cycle: builders start with a baseline classifier (e-small-LoRA), then progressively strengthen architectures in response to demonstrated attacks.

Dataset construction: Builders create two independent datasets to avoid data leakage. The baseline dataset (bld_data) draws from PHEME (human-authored misinformation) and Constraint 2021 (COVID-19 dataset), yielding 1,296 human-authored posts. Machine-generated posts are created via tweet injection — human posts are embedded into persona-engineered prompts fed to four instruction-tuned LLMs (Grok-4.1, Llama 3.3, GPT-4o, DeepSeek-R1). For the DASS variant, a paraphrased subset is generated via persona-controlled rewriting on a separate set of prompts.

Adversarial transformations: Breakers apply four technique families: (A) character-level (spacing edits, punctuation, random newlines); (B) lexical (paraphrasing via LLaMA 3.1, back-translation via intermediate languages, revised paraphrasing); (C) stylometric camouflage (emoji-based emotion assignment); (D) prompt-based evasion (tone, persona, rhetorical framing alterations). Transformations are combined into chained attacks, tested incrementally over five iterations. Breakers measure attack success via label-flip rate (LFR) — the fraction of initially correct predictions that become incorrect after transformation.

Detector architectures: The baseline uses e-small-LoRA fine-tuned on labeled data. Contrastive models employ siamese networks and triplet networks to cluster semantically similar texts while separating human-authored and machine-generated posts. The DASS variant extends triplet networks by dynamically selecting the anchor sample, forcing the model to learn broader stylistic features rather than shortcuts.

Semantic preservation: An automatic analysis pipeline computes NLI entailment scores, token-length ratio, ES-cosine similarity, and BLERT estimates to ensure adversarial variants preserve the original claim. Manual evaluation (three human annotators per sample) assesses semantic preservation, naturalness, and whether the misleading narrative is retained.

Results

Overall breakers' effectiveness: Across all 1,440 breaker configurations in five iterations, average detector accuracy declined from 71.1% (iteration 1) to 42.4% (iteration 5), while label-flip rate increased from 28.9% to 42.4%. By iteration 3, chained attacks of techniques {B1, D15, D23, B2, AR} reached LFR 85.6%, reducing detection accuracy to 14.4%. In iteration 5, the strongest configuration {B3, D13, D43, B2, AR} achieved 95.2% LFR and 4.8% accuracy.

Best-performing adversarial techniques: Back-translation to Arabic (B2_AR) was the single strongest technique, achieving 36% LFR across iterations. Persona-based prompt evasion variants (D2, D3, D4) evolved to higher effectiveness over iterations: D2 achieved 21.6% LFR; D3 reached 32.0%; D43 improved to 34.4% LFR. Chained combinations substantially outperformed isolated techniques, with combinations like {D2, B2} reaching 38.48% mean LFR across 80 configurations.

Detector robustness: Builders' models progressively strengthened in response to breaker techniques. The baseline e-small-LoRA achieved 100% accuracy on clean bld_data. Siamese networks (1:1 pairing) showed promise but TF-IDF-based hard negatives proved insufficient. Triplet networks with TF-IDF pairing achieved F1 0.999 on TURING TEST benchmarks. The DASS variant proved most robust, achieving an average accuracy of 72.68% after iteration 3, surpassing the prior iteration's best model (72% accuracy after iteration 2) by maintaining focus on stylistic features rather than surface-level keywords.

Iterative vs. static evaluation: Iterative breaker-builder rounds exposed substantially different detector vulnerabilities than single static evaluations. Results stabilized over iterations, with iteration 3-5 differences in median LFR shifting from 23.0% to 36.4% to 41.6%, demonstrating that static one-off testing substantially underestimates attack surface under adaptive adversarial conditions.

Manual evaluation findings: Semantic preservation analysis on 780 manually evaluated posts showed that adversarial transformations successfully preserved meaning: 85.6% of breaker-transformed posts received "claim held" or "claim partially held" scores. Naturalness ratings varied, with prompt-based evasion scoring higher (more natural) than character-level perturbations. The iterative framework revealed that semantic preservation is necessary but not sufficient — some highly preserving attacks still evade detection, indicating that detectors may learn spurious stylistic shortcuts rather than semantic understanding.

Connections

• Adversarial learning for fake news detection — core methodological foundation for testing detector robustness via iterative attack-defense cycles.
• LLM-generated text detection — directly addresses detection of LLM-generated social media content under adversarial manipulation.
• LLM-Generated Misinformation — applies adversarial techniques to disinformation narratives specifically.
• Machine-generated text detection — broader detector evaluation and benchmarking context.
• Semantic Preservation — key evaluation dimension showing transformation-preserving claims.
• Neural Network Architectures — discusses siamese, triplet, and dynamic anchor switching (DASS) architectures.
• Content-based fake news detection — social media disinformation detection baseline approaches.

Notes

Strengths: The paper addresses a critical gap in benchmarking LLM-generated content detectors: existing evaluations are static, not accounting for adaptive adversarial evolution. The BiBiR framework realistically models this arms race. The iterative methodology is innovative and likely to inform future adversarial robustness evaluation in NLP. Comprehensive adversarial technique taxonomy and the semantic preservation pipeline offer reusable tools. Two new datasets (bld_data, brk_data) are positioned for public release.

Limitations: The evaluation focuses specifically on Twitter/X social media posts with disinformation narratives, limiting generalization to other domains (news articles, forums, etc.). Manual evaluation was limited to selected samples (780 posts) rather than full evaluation. The persona-based generation uses a relatively small set of five templates — real-world persona diversity may be higher. The semantic preservation metrics (NLI, BLERT) themselves may have limitations in detecting subtle claim drift. The paper does not discuss compute costs or scalability of the iterative evaluation.

Follow-up questions: (1) How do results generalize to other social media platforms (Reddit, TikTok, Instagram)? (2) Do detector improvements from DASS architecture transfer to other adversarial domains (e.g., toxic comment detection)? (3) Can iterative evaluation be automated further to reduce human annotation burden? (4) How sensitive are results to the specific LLMs used for generation versus architect choices in detector design?

Related work: This work directly extends the Build it, Break it, Fix it (BiBiFi) framework from prior work and adapts it specifically to machine-generated disinformation on social media. The semantic preservation pipeline builds on prior work by Zellers et al. (Grover) and others on measuring whether text transformations preserve meaning. The use of siamese and triplet networks for detection follows prior approaches in zero-shot detection but applies them in an iterative adversarial context for the first time in this domain.