Skip to content
VLDBench: Evaluating Multimodal Disinformation with Regulatory Alignment

VLDBench: Evaluating Multimodal Disinformation with Regulatory Alignment

Authors: Shaina Raza, Ashmal Vayani, Aditya Jain, Aravind Narayanan, Vahid Reza Khazaie, Syed Raza Bashir, Elham Dolatabadi, Gias Uddin, Christos Emmanouidis, Rizwan Qureshi, Mubarak Shah

Institutions: Vector Institute for Artificial Intelligence (Toronto), University of Central Florida, Independent Researcher (USA), Toronto Metropolitan University, York University, University of Groningen

ArXiv: 2502.11361

TL;DR

VLDBench is the first large-scale human-verified benchmark for multimodal disinformation detection, comprising 31,339 news articles paired with images across 13 topical categories. Evaluation of 9 LLMs and 10 VLMs shows vision-language models substantially outperform text-only baselines (e.g., LLaMA-3.2-11B-Vision: 74.82% accuracy), but remain vulnerable to cross-modal attacks combining text and image manipulation. The benchmark is explicitly aligned with AI governance frameworks to support accountability and responsible deployment.

Contributions

  • VLDBench benchmark: A human-verified multimodal disinformation dataset containing 31,339 unique news articles (62,678 total instances: 31,339 text-only and 31,339 text-image pairs) curated from 58 news outlets across 13 categories (National, Business & Finance, International, Entertainment, Local/Regional, Opinion/Editorial, Health, Sports, Politics, Weather & Environment, Technology, Science, Other). All labels undergo rigorous expert verification with Cohen's κ = 0.78.

  • Semi-automated annotation pipeline: GPT-4o-assisted labeling combined with expert validation by 22 domain experts (500+ hours), achieving 92% raw agreement on audit subset. Topical categorization and binary disinformation labeling follow explicit guidelines distinguishing intentional manipulation from misinformation (false information shared without intent to deceive).

  • Comprehensive multimodal evaluation: Benchmarking of 9 language-only LLMs and 10 vision-language models, revealing vision-language models consistently outperform text-only baselines with accuracy gains of 5–30 percentage points. LLaMA-3.2-11B-Vision achieves best overall accuracy (74.82%).

  • Robustness analysis: Systematic evaluation under four perturbation types (text synonym substitutions/typos; image noise/blur; cross-modal mismatches; combined distortions) showing VLMs remain competitive under single-modality attacks but degrade significantly under cross-modal contradictions and combined distortions.

  • AI governance alignment: Mapping of benchmark components to MIT AI Risk Repository risk areas. Introduction of a Risk Scorecard with four lightweight governance indicators: Disinformation Risk (coverage across sensitive domains), Robustness Risk (degradation under stress conditions), Domain Shift Risk (out-of-distribution generalization), and reporting/transparency.

Method

Dataset Collection: The dataset spans May 2023 to September 2024, collected via Google RSS feeds from 58 outlets including Financial Times, CNN, New York Times, Wall Street Journal, and regional/international news sources. Entries include full article text and lead image; preprocessing removes low-resolution or media-centric URLs and enforces minimum 20-sentence articles to ensure textual depth.

Annotation Pipeline:

  1. Topical Categorization: GPT-4o assigns each article to one of 13 news categories based on content.

  2. Disinformation Labeling: GPT-4o evaluates each article as "Likely Disinformation" (when a verifiable false or manipulated claim is present and used persuasively) or "Unlikely Disinformation" (factually accurate or insufficiently manipulated to meet the disinfo threshold).

  3. Domain Expert Validation: 22 experts from computer science, linguistics, and journalism systematically reviewed all 62k samples. Disagreements resolved via consensus or expert adjudication. Multi-rater Fleiss' κ = 0.76 on stratified 500-sample audit subset confirms substantial inter-rater reliability.

  4. For VLM labeling: GPT-4o receives both text and image, ensuring labels reflect joint semantics rather than isolated modality; this captures cases where visual context (outdated or misattributed imagery) contradicts or misleadingly supports textual claims.

Task Definition: Binary classification: given a news article (text only or text + image), predict Likely vs. Unlikely Disinformation and provide a brief rationale. For multimodal inputs, detection is treated as binary classification over text-image pairs.

Evaluation Metrics: - Accuracy, Precision, Recall, F1-Score for binary labels - Robustness reported as relative performance drop (ΔF1) under perturbations - Reasoning quality assessed by human judgment: Prediction Correctness (0–5 scale, alignment with ground truth) and Reasoning Clarity (0–5 scale, interpretability).

Robustness Perturbations: - Text: Synonym substitution, misspelling (20% of words), negation flip - Visual: Gaussian blur, additive noise, resizing (±50%) - Cross-modal: Swapped imagery with same high-level category but contradictory detail; contradictory caption-image pairing - Combined: Simultaneous text and image distortion

Results

Zero-shot Performance (LLMs vs. VLMs):

Vision-language models substantially outperform text-only baselines: - Best VLM: LLaMA-3.2-11B-Vision, 74.82% accuracy, 72.45% F1 - Best LLM: LLaMA-3.2-1B, 70.29% accuracy, 69.35% F1 - Median improvement: +5 to +30 percentage points depending on model family

Models like LLaVA-v1.6-Mistral-7B and DeepSeek-VL2 also show strong performance (72.3–73.2% accuracy), while smaller models (Phi-3-mini, Vicuna-7B) show lower baselines (57.1–64.2%) but consistent gains from multimodality.

Instruction Fine-Tuning (IFT):

IFT consistently improves performance across architectures (+2–10 percentage points F1). For example: - Phi-3-mini-128k-Instruct: 67.2% F1 (zero-shot) → 72.6% (IFT) (+5.4pp) - LLaMA-3.2-11B-Vision: 72.45% F1 (zero-shot) → 75.9% (IFT) (+3.45pp)

Robustness Under Adversarial Perturbations:

VLMs degrade more sharply under cross-modal and combined attacks than text-only attacks: - Single-modality text perturbations: LLMs show ~3–5pp absolute drop; VLMs show 2–6pp drop - Cross-modal mismatches: VLMs drop 8–12pp; LLMs drop <2pp (text-only task mitigates visual noise) - Combined (text + image) attacks: VLMs drop 10–12pp; LLMs unaffected (text-only)

Example: LLaMA-3.2-1B-Vision original F1 = 72.45%; under combined attacks F1 = 62.03% (−10.4pp). This reveals vulnerability: while multimodality improves clean accuracy, inconsistent or mismatched modalities severely harm detection.

Domain Shift: Out-of-distribution evaluation on r/Fakeddit (Reddit fake/true binary classification) shows VLMs remain stronger than LLMs (71.40% vs. 66.20% accuracy) but degrade relative to in-domain performance (Δ = −3.4pp for VLMs vs. −0.9pp for LLMs), indicating cross-domain generalization challenges.

Human Evaluation: On 500-sample subset, LLaMA-3.2-11B-Vision achieves 75.2% prediction correctness and 3.5/5 reasoning clarity, outperforming LLaVA-v1.6 (72.0%, 3.0/5), suggesting that larger, more recent models produce more coherent disinformation rationales.

Connections

  • Extends Hameleers et al. 2020 on multimodal manipulation detection to the LLM/VLM era
  • Builds on benchmark methodology from FEVER and LIAR in dataset curation and annotation quality assurance
  • Related to robustness evaluation frameworks in Self-RAG and FineFact for fact verification under distribution shift
  • Complements Zhou et al. 2020 on multimodal fake-news detection by offering a large-scale governance-aligned benchmark
  • Aligns with AI safety frameworks discussed in Lazer et al. 2018 on the role of platforms and AI in information integrity

Notes

Strengths: - First large-scale, human-verified multimodal disinformation benchmark aligned with AI governance standards - Comprehensive robustness evaluation revealing systematic weaknesses in VLM cross-modal reasoning - High-quality annotation pipeline with expert consensus and transparent agreement metrics (κ = 0.78) - Practical findings: vision-language models substantially outperform text-only baselines, but gain is contingent on consistent multimodal input - Open release under CC-BY-NC-SA 4.0 supports reproducibility and community research

Limitations and Future Work: - Collection limited to May 2023–September 2024; temporal generalization unclear - Outlet diversity skews toward English-language North American and international mainstream sources; regional and non-English coverage is limited - Binary classification masks nuance: many real-world disinformation cases involve partial truth, context collapse, or subtle persuasion — not binary clear falsehood - Cross-modal attacks expose a critical gap: while multimodality improves accuracy on clean data, it introduces vulnerability to mismatched or contradictory visual-textual pairs — suggesting future work should focus on explicit multimodal alignment and consistency checking - Domain shift results on r/Fakeddit suggest in-domain overfitting; evaluation on longer-tail, emerging disinformation categories would strengthen the benchmark