Contributions
- Multilingual cross-domain dataset: 5,182 fact-checked articles in 40 languages spanning 105 countries, addressing the critical lack of multilingual COVID-19 misinformation resources during the early pandemic.
- Systematic aggregation pipeline: Collection from 92 fact-checking websites via Poynter and Snopes, with documented data cleaning (malformed URLs, missing titles, duplicate removal).
- Manual annotation into 11 categories: Explicit taxonomy of COVID-19 misinformation types—origins & spread, prevention & treatments, conspiracy theories, business & industry, humanitarian response, etc.—with high inter-annotator agreement (91–96%).
- Baseline classification model: Binary false/other classifier using BERT without fine-tuning, achieving practical utility for early-stage claim screening.
Method
Data collection: The authors systematically scrape fact-checked articles from Poynter (which indexes 91 organizations across 40 languages) and Snopes, extracting metadata including URLs, titles, publication dates, source outlets, countries of circulation, and embedded social media links. Language detection uses langdetect; content extraction via Beautiful Soup crawls HTML to retrieve article text and metadata.
Data cleaning: Faulty URLs are manually reviewed and corrected or removed; missing titles are sourced from webpage inspection; duplicates are filtered by URL. Preprocessing applies standard NLP techniques (lowercase, short-word removal, tokenization, abbreviation/contraction handling, spelling correction via TextBlob).
Annotation: Three trained annotators—fluent in English, Hindi, and German—independently label articles into 11 categories following a shared guideline. Inter-coder reliability is computed per language: 91% (English, n=2,216), 96% (Hindi, n=114), 94% (German, n=70).
Classification: A BERT-based model without fine-tuning is trained on English articles (2,116 false, 500 other) with hyperparameters: hidden units = 300, epochs = 150, batch size = 1, learning rate = 0.001. Training stops when validation loss plateaus.
Results
Classification performance (English test set):
- False class: precision 0.58, recall 0.75, F1 0.65
- Other class: precision 0.87, recall 0.75, F1 0.80
- Overall F1: 0.76
The authors emphasize the practical purpose: filtering false claims from a stream of candidate articles to reduce manual fact-checking labor, where recall on the false class is prioritized.
Dataset composition:
- Top countries: India (1,083 articles), USA (677), Spain (426), Brazil (283), France (229)
- Top fact-checker: AFP Fact Check (500+ articles)
- Social media links: 3,708 articles (71.5%) embed references to social platforms; Facebook most common (1,990, 38.4%), followed by Twitter (916, 17.7%), YouTube (602, 11.6%)
Notes
Strengths:
- Addresses critical gap: existing datasets (LIAR, FakeNewsNet) were English-only and not focused on pandemic misinformation.
- Aggregates from multiple independent fact-checkers, reducing outlet-specific bias.
- High inter-annotator agreement (91–96%) suggests reliable annotations.
- Public GitHub repository and explicit data schema enable reproducibility and follow-up work.
- Systematic collection methodology explicitly described.
Limitations:
- Only 1,951 of 5,182 articles (37.5%) manually annotated for category; remaining rely on labels from fact-checking websites (potentially inconsistent across outlets).
- Binary classification (false vs. other) is pragmatic but coarse; 23 original fact-check classes are flattened to binary.
- BERT-without-fine-tuning establishes a weak baseline; the paper acknowledges stronger results would be possible.
- Dataset frozen at May 15, 2020; COVID-19 misinformation narratives and topics shifted significantly in subsequent waves.
- Limited language annotation (only English, Hindi, German manually categorized despite 40-language dataset).
Fit: Foundational dataset paper for COVID-19 misinformation, directly in GUIDE's scope. The multilingual approach and systematic aggregation methodology have influenced subsequent work in pandemic-era fake news detection.