Contributions
- Formulate the novel task of detecting whether a claim was previously fact-checked, distinct from but complementary to end-to-end fact-checking.
- Release two specialized datasets: PolitiFact (768 input–verified pairs) and Snopes (1,000 input–verified pairs with 10,396 verified claims), using naturally occurring claims from political debates and social media rather than normalized claims.
- Propose learning-to-rank models that achieve substantial improvements (MRR up to 0.605 on PolitiFact, 0.778 on Snopes) over state-of-the-art retrieval and semantic similarity baselines, and demonstrate that sentence-BERT representations outperform standard BERT.
Method
The authors frame the task as a ranking problem: given a check-worthy input claim and a collection of previously fact-checked verified claims, rank the verified claims so that claims useful for verifying the input are ranked highest.
Datasets: PolitiFact is sourced from the fact-checking website PolitiFact, which fact-checks claims made by politicians and public figures; verified claims are extracted alongside their fact-checked labels (true/false/other). Snopes is derived from tweets paired with Snopes fact-check articles. Input claims are sentences from political debates or tweets; verified claims are the previously fact-checked counter-parts. The datasets differ from prior normalized fact-checking datasets (e.g., LIAR) in that they preserve the naturally occurring language variability of real claims.
Baselines and models:
- BM25: Classical IR baseline using TF-IDF weighted cosine similarity on exact keyword matches. Tested against claim text, verified claim text, article title, and article body.
- BERT-based semantic matching: Encode input and verified claims using pre-trained BERT variants (base, RoBERTa, sentence-BERT) and compute cosine similarity. Sentence-BERT (trained to produce sentence-level representations) outperforms standard BERT.
- Learning-to-rank: Combine signals from BM25 and semantic similarity using rankSVM with an RBF kernel. Given top-N results from IR, the ranker reorders them by learning a pairwise preference function over embeddings and exact-match scores.
Results
On PolitiFact (testing on 154 input–verified pairs):
- BM25 baseline (title only): MRR 0.288
- Sentence-BERT (VerClaim representation): MRR 0.435
- Combining IR, semantic similarity, and article body: MRR 0.538
- Learning-to-rank on top-200: MRR 0.605, MAP@20 0.590
On Snopes (testing on 200 input–verified pairs):
- BM25 baseline (title only): MRR 0.619
- Sentence-BERT variants: MRR 0.543–0.655 depending on representation choice
- Learning-to-rank on top-200: MRR 0.778
The re-ranking step provides consistent absolute improvements of 0.04–0.05 points MRR across both datasets, and performance degradation is minimal even when searching against larger verified-claim collections (10,396 for Snopes).
Notes
This paper addresses a practical bottleneck in fact-checking: before spending effort verifying a new claim, check if a trusted organization has already done the work. The focus on naturally occurring claims in political contexts (debates, social media) rather than normalized datasets increases applicability to real-world fact-checking workflows. The learning-to-rank formulation elegantly combines the complementary strengths of exact matching and semantic similarity. A potential limitation is the reliance on manual annotation of input–verified pairs; scaling to broader claim domains or languages remains open.