Reliable training data is the foundation of misinformation research. Data collection encompasses source selection, content acquisition, human annotation for ground-truth labels, quality assurance, and documentation of collection methodology and potential biases.
Collection challenges
- Reliability of labels: Human annotators often disagree on whether a claim is false, partially true, or true. Defining veracity ground truth is itself a research problem, particularly for claims that are context-dependent or admit multiple interpretations.
- Source bias: Using fake news outlets as the negative class (e.g., FakeNewsNet) conflates content-level falsity with source-level reputation, encoding publisher identity rather than veracity signals into classifiers.
- Topic imbalance: Most fake news datasets are heavily skewed toward politics, elections, and health misinformation, limiting generalization to other domains.
- Scale and cost: Reliable annotation requires expert effort or multiple annotators per instance for consensus. Large balanced datasets are expensive to produce.
- Temporal drift: Narratives, tactics, and platform affordances change rapidly; datasets collected in 2017 may not reflect 2024 disinformation strategies.
- Privacy and consent: Scraping social media posts or collecting user data raises ethical and legal concerns.
Collection strategies
- Crowdsourcing: Amazon Mechanical Turk, Appen, or similar platforms for large-scale cheap annotation; prone to quality issues but enables scale.
- Expert annotation: Fact-checkers, journalists, or researchers manually verify claims; high quality but low scale.
- Hybrid approaches: Use crowdsourcing with expert adjudication on disagreements; balances cost and quality.
- Web scraping fact-checking sites: Snopes, PolitiFact, and similar sites provide labeled claims with explanations; leverages existing expert effort but may bias toward certain narrative types.
- Synthesizing labels from multiple sources: Combine labels from different fact-checking sites or use voting schemes to reduce individual source bias.
Key papers
- 2019-asr-big-data-quality-data-fake-news — Analyzes quality and balance challenges in existing fake news datasets; introduces MisInfoText repository (a public collection of news articles labeled for veracity) collected via scraping and manual assessment of fact-checking sites; calls on community to develop standards for data collection and quality assurance.
- 2025-thibault-guide-misinformation-datasets — Comprehensive survey of 75 misinformation datasets with quality assessment framework; identifies spurious correlations, temporal drift, and label-quality issues; proposes Evaluation Quality Assurance (EQA) framework for improving dataset standards.
Related topics
- Datasets and benchmarks — the products of collection effort
- annotator-disagreement — how inter-rater agreement and disagreement affect label quality
- text-classification — methods trained on collected data
Notes
Lessons learned: The most useful datasets (LIAR, FakeNewsNet, CoAID) have become bottlenecks: widespread reuse creates dataset bias where models overfit to specific label artifacts or source biases rather than learning generalizable signals. Future research should diversify sources (geography, language, platform, topic) and invest in transparent documentation of collection methodology and known limitations.