COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic

Contributions

• Semi-automatic COVID-19 fact-checking dataset: 4,086 real-world claims (1,407 supported, 2,790 automatically generated refuted claims) sourced from the r/COVID19 subreddit with linked evidence documents and sentence-level annotations.
• Automatic claim selection pipeline: Filters subreddit posts for well-formedness, trustworthiness (upvote ratios, source credibility), and domain relevance using syntactic heuristics and source-ranking signals.
• Automatic counter-claim generation: Two-step approach using attention-based salience word selection (via BERT fine-tuned on SciFact) and masked language model infilling (RoBERTa) to generate plausible negations without manual annotation.
• Evidence sentence selection via crowdsourcing: Uses SBERT sentence embeddings to rank candidate evidence sentences, selects top-5 candidates, and validates with Amazon Mechanical Turk workers; achieves moderate inter-annotator agreement (Cohen's κ = 0.47) on evidence relevance.
• Benchmark evaluation: End-to-end pipeline (Google Search + SBERT for evidence retrieval, RoBERTa for veracity prediction) establishes baseline performance; demonstrates that models trained on domain-specific data improve over zero-shot transfer.

Method

Claim and Trustworthy Evidence Selection

The pipeline starts with the heavily moderated r/COVID19 subreddit. Each post is filtered to ensure: (1) well-formed claims (full sentences, not questions); (2) high trustworthiness (upvote ratios > 0.7, sources with Alexa rank < 50,000, linked sources in top Google results); (3) relevance to COVID-19 topics (vaccines, therapies, epidemiology, policy).

Initial filtering from 22,616 posts yields 1,407 true claims with associated source evidence documents; an additional 111 claims are gathered from Google search results.

Automatic Counter-Claim Generation

Counter-claims are generated in two stages:

1. Salience word selection: Attention-based extraction from BERT classifiers fine-tuned on SciFact, identifying keywords whose reversal would negate the claim. For example, in "Oxford vaccine triggers immune response," the salience word "triggers" is identified.

2. Masked language model infilling: RoBERTa (fine-tuned on CORD-19) generates candidate counter-claims by replacing salience words with antonyms or negations. A contradiction score (RoBERTa on Multi-NLI) filters the top candidates; a threshold of 0.9 precision is used to select final counter-claims.

This process yields 1,042 high-quality counter-claims from the 1,407 true claims; 4,042 total claims after additional filtering for well-formedness.

Evidence Sentence Selection

For each claim, five potential evidence documents are retrieved via Google Search + SBERT ranking. Evidence sentences are selected by computing cosine similarity between SBERT embeddings of the claim and candidate sentences; crowdsourced workers (Amazon Mechanical Turk) then validate whether the sentence supports or refutes the claim.

Results

Dataset composition:

• Train: 1,036 supported, 2,227 refuted
• Dev: 130 supported, 289 refuted
• Test: 130 supported, 274 refuted
• Total: 1,407 supported, 2,790 refuted (4,086 claims)

Evidence retrieval performance:

• Top-5 retrieval: precision 22.27, recall 52.37, F1 31.25
• Top-3 retrieval: 24.77, 45.14, 31.99
• Top-1 retrieval: 29.68, 29.93, 29.80

The baseline pipeline (Google Search + SBERT) achieves F1 = 32.0, substantially outperforming prior fact-checking approaches on COVID-19 (FEVER baseline on COVID-Fact dev set: F1 = 18.26).

Veracity prediction (binary SUPPORTED / REFUTED):

• COVID-Fact (trained): 83.5% accuracy (claim-only), 82.0% F1 (top-5 evidence), 83.0% F1 (top-3), 81.0% F1 (top-1)
• SciFact + COVID-Fact (transfer): 82.2% accuracy (best result: 83.0% F1 with top-3 evidence)
• FEVER + COVID-Fact (transfer): 74.3% accuracy, 70.1% F1

Zero-shot transfer (models trained on SciFact/FEVER, tested on COVID-Fact):

• MNLI fine-tuned RoBERTa: 61.3% accuracy
• SciFact (Wadden et al., 2020): 56.9% accuracy
• FEVER (Thorne et al., 2018): 48.3% accuracy

The substantial gap between domain-specific training (83.5%) and zero-shot transfer (48–61%) highlights the vocabulary and claim-type divergence between general-purpose and COVID-specific datasets.

Notes

Strengths:

• Real-world claims: Sourcing from Reddit's r/COVID19 captures authentic natural-language claims rather than journalist-written synthetic claims, improving ecological validity.
• Automatic counter-claim generation: Eliminates the expensive manual annotation bottleneck; the two-step salience + infilling approach is creative and produces plausible negations (contradiction score threshold validates quality).
• Domain-specific challenge: Demonstrates a 35-point accuracy gap (83.5% vs. 48.3%) between domain-trained and zero-shot models, highlighting that general-purpose fact-checking datasets fail on specialized pandemic vocabulary (e.g., clinical terms, vaccine mechanisms).

Weaknesses:

• Moderate crowdsourcing agreement: Cohen's κ = 0.47 on evidence relevance indicates non-trivial subjectivity; workers struggled with nuanced evidence-claim alignment, especially for claims involving domain-specific knowledge (e.g., immunology).
• Limited evidence sources: Pipeline retrieves evidence from top-5 Google results + r/COVID19 posts; may miss authoritative but less-visible sources (clinical trials, preprints from less-indexed servers).
• Claim generation artifacts: Automatic counter-claim generation via salience word replacement risks introducing grammatical errors or semantically implausible negations (e.g., flipping verbs without preserving argument structure). The Multi-NLI fine-tuned score filtering helps but is not exhaustive.
• Temporal generalization: Claims were collected during 2020–2021; vaccine efficacy and variant evolution mean evidence and verdicts shift over time. Dataset stability for inference beyond 2021 is unclear.

Technical clarity issues:

• The paper notes that models trained on COVID-Fact claim-only (83.5%) outperform models given gold evidence (82.0% with top-5), a counterintuitive result. The authors attribute this to the model learning spurious correlations between claim wording and veracity rather than faithful evidence reasoning—a significant finding warranting deeper investigation.
• Evidence sentence selection uses both automatic SBERT ranking and crowdsourced validation; the interplay between automatic filtering (which may introduce bias) and human judgment deserves clearer analysis.

Significance:

The paper makes a clear contribution to fact-checking research by (1) introducing a domain-specific dataset that exposes zero-shot transfer gaps, (2) demonstrating automation of dataset construction (counter-claim generation without human writers), and (3) establishing baselines on a real-world pandemic claim corpus. However, the limited scale (4K claims vs. FEVER's 185K) and moderate inter-annotator agreement on evidence selection temper the impact. The dataset fills a niche but is more specialized than broadly applicable.