Skip to content
A Guide to Misinformation Detection Data and Evaluation

A Guide to Misinformation Detection Data and Evaluation

Authors: Camille Thibault, Jacob-Junqi Tian, Gabrielle Peloquin-Skulski, Taylor Lynn Curtis, James Zhou, Florence Laflamme, Luke Yuxiang Guan, Reihaneh Rabbany, Jean-François Godbout, Kellin Pelrine

Venue: KDD '25 — 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, August 3–7, 2025, Toronto, ON, Canada

DOI: 10.1145/3711896.3737437

TL;DR

This work surveys the largest collection of misinformation detection datasets to date (75 datasets from prior literature) and evaluates dataset quality across multiple dimensions. The authors assess whether datasets enable reliable veracity evaluation by analyzing keyword correlations, temporal correlations, and feasibility of claim assessment, finding that many datasets contain spurious correlations making label quality questionable. They introduce the Complete Data Lab Misinformation Database (CDL-MD), a unified collection of 75 datasets with standardized evaluation, and propose Evaluation Quality Assurance (EQA) as a framework for improving evaluation methodologies in the field.

Contributions

  • Largest dataset survey: Comprehensive analysis of 75 misinformation detection datasets, categorizing by task type (claim classification, paragraph detection), modality (text, image, video, audio), languages, geographic regions, and collection methods.

  • Data quality assessment framework: Systematic evaluation of dataset quality across three dimensions:

  • Spurious keyword correlations (many datasets show unrealistically strong predictive signals from isolated keywords)
  • Spurious temporal correlations (label quality degrades when temporal information is temporally separated from its context)

  • Feasibility of veracity assessment (many claims are too vague or ambiguous to reliably assess; over 50% of claims in 29 datasets lack sufficient evidence for veracity evaluation)

  • Labeling approach taxonomy: Categorizes how datasets establish ground truth: human expert annotation, crowd-sourced annotation, source-based categorization, algorithm-generated labels, and hybrid approaches; shows implications for label noise and generalization.

  • CDL-MD database: Unified collection of 75 datasets with standardized metadata, modality information, and quality assessments; integrated into HuggingFace infrastructure.

  • Evaluation Quality Assurance (EQA): Framework proposing that research should validate evaluation methodology alongside methodology design—including whether datasets are suitable for the evaluation task and whether metrics correctly measure performance.

Method

The authors conducted a systematic literature review of misinformation detection datasets, extracting dataset characteristics (size, modality, language, annotation method, label distribution). They then evaluated dataset quality using three key metrics:

Spurious Keyword Correlations: Using a random forest classifier trained on binary keyword presence/absence features (removing stop words and excluding tweets-only datasets), the authors compared predictive performance against a baseline of random prediction. Six datasets showed spurious keyword correlations with macro F1 scores exceeding 50 baseline threshold, indicating that models trained on these datasets may learn unrealistically strong spurious signals.

Temporal Correlations: For datasets with temporal information, the authors trained a random forest classifier using tweet ID digits or date components. Several datasets (Twitter15, Twitter16, MediaEval, Rumors, X-Fact) showed high temporal predictivity, suggesting that label quality degrades when temporal context is separated from the actual claim.

Feasibility Assessment: The authors analyzed claim feasibility by sampling 100 examples from each dataset and having 8 human annotators and 1 AI (GPT-4o with web search) assess whether each claim could be credibly evaluated for veracity. Claims were categorized as: - Feasible: Sufficient context for credible assessment - Feasible with Web Search: Some context missing but retrievable online - Not Feasible: Too vague, ambiguous, or requiring information that cannot be accessed

Results

Keyword Predictivity: Six datasets (CoAID, IFND, MM-COVID, NLPF2021, TruthSeeker2023, TI-CNN) showed macro F1 scores exceeding 50% in keyword-only classification, indicating potential for spurious learning. The authors recommend caution when training and testing models on these datasets.

Temporal Predictivity: Nine datasets showed temporal features with >50% macro F1 scores, suggesting that temporal information correlates with label independent of claim content. This is particularly problematic for datasets designed for early rumor detection.

Feasibility Evaluation: Results show that without evidence retrieval systems, at least half of claims in 29 datasets are infeasible to assess for veracity. When given web search access, feasibility improves substantially, with 75% of claims rated feasible in the most favorable cases. GPT-4o achieves comparable performance to human experts (0.56 Pearson correlation with human labels on feasibility predictions).

Label Quality and Annotation Methods: Datasets using expert annotation (professional fact-checkers, journalists, domain experts) generally have higher label quality than those using crowdsourced workers. However, even expert-annotated datasets show moderate inter-annotator agreement.

Generalization Insights: The paper demonstrates that standard evaluation metrics like accuracy and F1 are poor proxies for whether models will generalize to new data. Simpler metrics like baseline-relative performance provide better indications of dataset suitability.

Connections

  • Related to Zhou & Zafarani (2020) — A Survey of Fake News as the prior definitive survey of detection methods; this work complements it by focusing on data quality rather than methods.
  • Cites foundational work on the importance of annotation quality and expertise in classification tasks.
  • Extends conversations on identifying and assessing mislabeled datasets.
  • Shares methodological concerns with Guess et al. (2020) on generalization of detection systems across contexts and domains.
  • Relevant to any work using misinformation datasets for training or evaluation, particularly Chen et al. (2024) and other recent papers on LLM-based detection.

Notes

This survey fills a critical gap in the misinformation detection literature by moving beyond "more datasets = more progress" to a rigorous assessment of whether datasets are actually suitable for their intended use. The key finding—that many widely-used datasets contain spurious correlations making them unreliable for model evaluation—has significant implications for how researchers should design and report experiments.

The feasibility analysis is particularly impactful: the observation that over 50% of claims are difficult to assess for veracity even with context challenges the premise that fully-supervised detection is the right approach. This suggests that future work should focus on semi-supervised methods, confidence estimation, or techniques that make explicit when claims are too ambiguous to judge.

The proposed Evaluation Quality Assurance framework is valuable for the field. Standard practices around reporting data splits, metric selection, and cross-domain evaluation would benefit from formal assessment of whether the evaluation setup is valid for the research question being asked.

One limitation: the survey is necessarily time-bound (data collection through 2024) and covers only datasets the authors could locate; very recent or domain-specific datasets may be missing. The feasibility annotations themselves represent subjective judgments that could vary across annotators or domains.

Recommended as essential reading for anyone planning misinformation detection research, particularly those designing new datasets or selecting existing datasets for method evaluation.