Big Data and quality data for fake news and misinformation detection

Fatemeh Torabi Asr; Maite Taboada

doi:10.1177/2053951719843310

In brief

Addresses the critical problem of collecting large, quality, balanced labeled datasets for fake news detection. Introduces MisInfoText, a publicly available repository of news articles labeled for veracity, and analyzes topic distribution across existing fake news datasets to show how dataset imbalance affects text classification performance.

Contributions

Comprehensive review of fake news detection approaches from three perspectives: educating the public, manual fact-checking, and automatic classification
Introduction of MisInfoText repository: a GitHub-hosted collection of news article texts labeled for veracity
Analysis of topic coverage and distribution in available fake news datasets (Buzzfeed, Snopes, Emergent)
Empirical demonstration that topic distribution differs significantly between true and false news
Call to community to develop standard, quality-assured datasets for fake news research

Method

The paper reviews and consolidates publicly available fake news detection datasets and constructs the MisInfoText repository. The authors:

Data collection — Scraped and cleaned news articles from three sources: - Buzzfeed: 156 articles manually labeled false/true - Snopes: ~4000 articles with veracity labels and fact-check explanations - Emergent: 1612 articles labeled across a 5-way spectrum
Topic modeling — Applied Latent Dirichlet Allocation (LDA) to the news texts and projected the labeled documents into a topic space. Generated 10 major topics via LDA with fine-tuned hyperparameters.
Topic distribution analysis — Created heat maps showing the distribution of topics across veracity labels (true, false, mostly true, mostly false, unknown/mixture) for each dataset.
Comparative analysis — Examined which topics are more frequent in false news (politics, police, environment, technology) versus true news, and noted variation across datasets and fact-checking websites.

Results

Topic modeling identified systematic differences in topic prevalence: for example, in the Snopes dataset, politics and police topics are more frequent in false news, while in the Emergent dataset, technology and environment topics are more common in false news.
Topic imbalance is substantial across datasets: different topics dominate true versus false news in different corpora, making it difficult for classifiers trained on one dataset to generalize to new topic distributions.
The three datasets show varying topic coverage — Buzzfeed and Snopes are heavily biased toward U.S. politics and recent events, while Emergent has broader topical diversity but less balanced labeling.
Feature-based and deep learning text classification methods both struggle when training data is skewed by topic or when vocabulary/topic vocabulary differs significantly between training and test distributions.

Notes

Strengths: Comprehensive review of available datasets with clear documentation of collection methods and limitations. The topic modeling analysis is insightful and reveals a fundamental problem: classifiers learn dataset-specific biases tied to topic distribution, limiting their real-world generalization. The open-source MisInfoText repository has proven valuable for the research community.

Limitations: The paper identifies the problem of dataset imbalance and quality but does not propose a complete solution. The datasets analyzed are still relatively small by modern standards (under 2000 labeled instances in most cases). The paper calls for action but does not deliver a definitive high-quality dataset.

Follow-up work: Since 2019, the field has developed larger datasets (e.g., LIAR, FakeNewsNet) and begun addressing the topic/source bias problem, but the fundamental challenge of collecting diverse, reliably labeled fake news data remains.