CovidLies: Detecting COVID-19 Misinformation on Social Media

Tamanna Hossain; Robert L. Logan IV; Arjuna Ugarte; Sean Young; Yoshitomo Matsubara; Sameer Singh

In brief

Introduces CovidLies, a benchmark dataset of 86 COVID-19 misconceptions paired with 6,761 Twitter annotations for a two-stage misinformation detection task: misconception retrieval (which known falsehoods are relevant) and stance detection (whether tweets agree, disagree, or take no stance). Domain-adapted BERT models achieve best performance, with 61.3 H@1 retrieval and 50.2 macro F1 for stance classification.

Code & Data: GitHub

Contributions

CovidLies dataset: 86 COVID-19 misconceptions extracted from Wikipedia, 6,761 expert-annotated tweet-misconception pairs from March–April 2020, labeled with stance labels (Agree, Disagree, No Stance)
Two-stage formulation: Separates misinformation detection into misconception retrieval and stance detection, enabling fine-grained understanding of how tweets relate to specific false claims
Benchmark evaluation: Tests multiple NLP approaches (TF-IDF, BM25, semantic similarity, BERT-based models) and shows domain adaptation improves both stages
Public resource: Released dataset and code to support research on pandemic-related misinformation

Method

The task pipeline consists of two sub-tasks:

Misconception Retrieval. Given a tweet and a set of known COVID-19 misconceptions, identify which misconceptions (if any) are discussed in the tweet. Framed as a ranking problem where misconceptions are ranked by similarity to the tweet using TF-IDF, BM25, or learned embeddings (Cosine Similarity with GloVe, BERT).

Stance Detection. For each relevant misconception, classify the tweet's stance as Agree (tweet supports the misconception), Disagree (tweet contradicts it), or No Stance (tweet mentions it but takes no position). Formulated as three-way classification using supervised classifiers (linear models, BILSTM, SBERT) trained on related NLP datasets (SNLI, MultiNLI, MedNLI) and fine-tuned on the CovidLies data.

Models evaluated include non-contextualized embeddings (TF-IDF, BM25, GloVe), contextualized representations (BERT-LARGE, domain-adapted BERT trained on COVID-19 Twitter data), and combined approaches using BERTScore with domain adaptation plus NLI models for stance classification.

Results

Misconception Retrieval: Domain-adapted BERTSCORE achieves 61.3 H@1 (hits at rank 1), 92.7 H@5, 96.9 H@10, with MRR of 0.77. BM25 and non-adapted BERT perform worse, showing domain adaptation is critical for COVID-19 text.

Stance Detection: BERTSCORE (DA) + SBERT (DA) on MultiNLI achieves highest macro F1 of 50.2, with 41.2 F1 on Agree class and 55.9% precision on No Stance. Strong class imbalance (85% No Stance tweets) complicates detection; agreement-focused tweets are harder to classify than disagreement tweets.

Domain Adaptation Impact: Pre-training on COVID-Twitter-BERT significantly improves both retrieval and stance detection, especially on minority classes (Agree, Disagree).

Notes

Strong contribution for its timing—among the first to address COVID-19 misinformation at scale with NLP. The two-stage formulation (retrieval + stance) is cleaner than end-to-end classification and allows fine-grained error analysis. Dataset remains in-progress (not all 86 misconceptions fully annotated), limiting generalization. Heavy class imbalance (No Stance tweets dominate) reflects real-world distribution but makes minority-class detection challenging. Benchmark results show existing NLP models struggle with domain shift; more work needed to move beyond domain-specific pretraining.