Skip to content

WildFakeBench

WildFakeBench is a large-scale benchmark dataset for micro-video misinformation detection, comprising 10,107 real-world videos collected from six social platforms (YouTube, Twitter, Facebook, TikTok, Instagram, Kuaishou) spanning 2017–2025.

Dataset characteristics

  • Scale: 10,107 videos with 4,122–5,985 real videos and 1,991–10,107 total misinformation samples
  • Sources: YouTube, Twitter, Facebook, TikTok, Instagram, Kuaishou (covering English and Chinese content)
  • Temporal coverage: 2017–2025, capturing evolving misinformation tactics
  • Annotation levels:
  • Coarse: Real vs. fake binary classification
  • Fine-grained: Four primary deception categories with 11 subtypes:
    1. Manipulation (3 subtypes): Text Tampering, Video Tampering, Audio Tampering
    2. Biases (3 subtypes): Faulty Logic, Exaggerated Narrative, Offensive Content
    3. AIGC (2 subtypes): AI Edition, Deepfake
    4. Out-of-Context (3 subtypes): Knowledge Error, Event Fabrication, Event Splicing

Annotation process

  • Expert verification: Videos verified against fact-checked sources (PolitiFact, China Internet Joint Rumor Refuting Platform) to ensure ground truth
  • Multi-annotator consensus: Each sample independently annotated by 3+ experts; final labels determined by majority vote
  • Consistency checks: Duplicate removal via textual similarity filtering; topic diversity maintained across categories

Modalities

  • Video frames: 8 frames uniformly sampled per video
  • Audio: Full audio track with transcription
  • Text: Video titles, captions, and overlaid text
  • Metadata: Platform, upload date, creator information (where available)

Evaluation metrics

The dataset enables evaluation across: - Multi-class micro-accuracy: Per-category classification accuracy - Macro-level metrics: F1 score, precision, recall per deception type - Explanation quality: Human assessment of fine-grained attribution explanations - Cross-platform generalization: Transfer learning across platform-specific misinformation patterns

Access and licensing

Dataset released under Attribution NonCommercial License (CC BY-NC-SA 4.0). Code and dataset available at https://github.com/Aiyidan/FakeAgent.

Benchmarks and baselines

Benchmark provides evaluation of: - Closed-source LLMs: GPT-4o, Qwen2.5-VL-32B, etc. - Open-source models: InterVL-2.5-38B, VideoLLaMA2-7B, etc. - Proposed method: FakeAgent (68.63% micro-accuracy)

Detailed baseline performance reported in From Manipulation to Mistrust: Explaining Diverse Micro-Video Misinformation.

  • FakeSV: 1,827–1,991 videos from Twitter/YouTube on COVID-19
  • Extended FakeNewsNet: multimedia articles extension of FakeNewsNet
  • FakeRecipe: 1,172–1,991 videos from TikTok and Twitter