Skip to content

VLDBench

Vision-Language Disinformation Benchmark: A large-scale, human-verified multimodal dataset for evaluating disinformation detection systems against both text-only and multimodal (text + image) inputs.

Dataset overview

  • Size: 31,339 unique news articles, each paired with an image (lead image). Total: 62,678 instances (31,339 text-only, 31,339 text-image pairs).
  • Sources: 58 news outlets including Financial Times, CNN, New York Times, Wall Street Journal, Reuters, Washington Post, and regional/international sources.
  • Topical categories: 13 categories with multi-label assignment (percentages may exceed 100%):
  • National, Business & Finance, International, Entertainment, Local/Regional, Opinion/Editorial, Health, Sports, Politics, Weather & Environment, Technology, Science, Other
  • Collection period: May 2023 to September 2024.
  • Task: Binary classification: Likely Disinformation (verifiable false or manipulated claim present and used persuasively) vs. Unlikely Disinformation (factually accurate or insufficiently manipulated).

Annotation process

  1. LLM-assisted categorization: GPT-4o assigned each article to topical categories.
  2. LLM-assisted labeling: GPT-4o evaluated binary disinformation label with explicit rationale.
  3. Expert validation: 22 domain experts (computer science, linguistics, journalism) systematically verified all 62,678 samples across 500+ hours.
  4. Quality: Cohen's κ = 0.78 on full dataset; Fleiss' κ = 0.76 on stratified 500-sample audit subset.
  5. Disagreement resolution: Consensus or expert adjudication.

Key statistics

Metric Value
Number of news sources 58
Number of categories 13
Modalities Text, Image
Unique articles (paired text + image) 31,339
Annotated instances (text-only) 31,339
Annotated instances (multimodal: text + image) 31,339
Total annotated instances 62,678
Average article length 1,168.7 words
Image-caption coverage 100% (avg. 26.7 words per caption)
Likely Disinformation 57.1%
Unlikely Disinformation 42.9%
Collection period May 2023 – September 2024

Evaluation benchmarks

The dataset has been evaluated against:

Language-only LLMs: - Phi-3-mini-128k-Instruct (57.15±0.29% accuracy) - Vicuna-7B-v1.5 (55.21±0.40%) - Mistral-7B-Instruct-v0.3 (68.58±0.45%) - Qwen2-7B-Instruct (69.92±0.50%) - Intern-LM2-7B (51.19±0.32%) - DeepSeek-V2-Lite-Chat (51.96±0.49%) - GLM-4-9B-Chat (51.14±0.55%) - LLaMA-3.1-8B-Instruct (68.21±0.42%) - LLaMA-3.2-1B-Instruct (70.29±0.36%)

Large Vision-Language Models (VLMs): - Phi-3-Vision-128k-Instruct (64.18±0.43% accuracy) - LLaVA-v1.5-Vicuna-7B (72.32±0.48%) - LLaVA-v1.6-Mistral-7B (72.38±0.42%) - Pixtral (70.18±0.39%) - Qwen2-VL-7B-Instruct (67.28±0.33%) - InternVL2-8B (63.57±0.36%) - DeepSeek-VL2-Small (68.15±0.31%) - DeepSeek-Janus-Pro-7B (70.04±0.42%) - GLM-4V-9B (62.13±0.44%) - LLaMA-3.2-11B-Vision (74.82±0.39%) — best overall accuracy

Robustness evaluation

The dataset includes evaluation under adversarial perturbations:

Perturbation types: - Text: Synonym substitution, misspelling (20% of words), negation flip - Visual: Gaussian blur, additive noise, resizing (±50%) - Cross-modal: Swapped imagery (high-level category match, but contradictory detail); contradictory caption-image pairing - Combined: Simultaneous text and image distortion

Key finding: VLMs degrade sharply under cross-modal mismatches (8–12pp F1 drop) and combined attacks (10–12pp drop), while remaining robust to single-modality text perturbations. This reveals a critical vulnerability: while multimodality improves accuracy on clean data, inconsistent or mismatched modalities severely harm detection.

Access

  • Data & code: https://huggingface.co/datasets/vector-institute/VLDBench and https://github.com/VectorInstitute/VLDBench
  • License: CC-BY-NC-SA 4.0

Notes

VLDBench is notable for: 1. Large scale: 31,339 articles with 500+ hours of expert annotation, one of the highest-quality multimodal disinformation benchmarks to date. 2. Governance alignment: Explicitly mapped to MIT AI Risk Repository, enabling evaluation against policy-relevant risk dimensions. 3. Multimodal dual-task: Supports both text-only and text-image evaluation, enabling direct comparison of unimodal vs. multimodal detection approaches. 4. Robustness focus: Systematic perturbation evaluation reveals architectural vulnerabilities in VLMs to cross-modal attacks. 5. Domain diversity: 58 outlets across 13 categories provide broader coverage than politicized datasets (e.g., PolitiFact/GossipCop).

Limitations: Collection limited to May 2023–September 2024; English-language sources only; outlet diversity skews toward North American and international mainstream media.