The Quest to Automate Fact-Checking

Contributions

• Defines what a fully automated fact-checking system would require: automated claim detection, instant verification, accuracy equal to or exceeding human fact-checkers, and transparent documentation of sources and reasoning
• Articulates the major computational challenges: extracting claims from speech/text, separating factual claims from opinions, determining check-worthiness, collecting sufficient evidence, and monitoring evolving claims
• Identifies journalistic barriers: lack of structured data in fact-checking publications, absence of centralized repositories, inconsistent transparency, and time-intensive manual claim identification
• Presents ClaimBuster, a practical tool that learns to rank sentences by check-worthiness using supervised classification on debate transcripts
• Demonstrates ClaimBuster on the 2015 Republican presidential debate, showing strong agreement with professional fact-checkers' selection of claims

Method

ClaimBuster approach:

ClaimBuster models claim detection as a supervised classification and ranking task. Sentences are categorized into three classes:

• Non-Factual Sentences (NFS): Opinions, beliefs, declarations, questions without factual content
• Unimportant Factual Sentences (UFS): Factual but not check-worthy—not of public interest
• Check-worthy Factual Sentences (CFS): Factual claims the public should know about

A Support Vector Machine classifier is trained on 8,015 labeled sentences from past presidential debates to output a posterior probability for each sentence that it is a CFS. Sentences are ranked by this probability score.

Features: The model uses 6,201 features extracted from sentences:

• Sentiment scores (negative to positive)
• Length (word count)
• Word-level TF-IDF features
• Part-of-speech tags (43 POS categories; cardinal numbers proved most discriminative)
• Named entity types (26 types extracted via AlchemyAPI)

Feature selection via random forest reduced to top ~100 most important features. Cardinal numbers (CD tag) emerged as the strongest signal—check-worthy claims contain numeric facts at higher rates than non-factual sentences.

Results

Classification performance on 8,015 sentences:

• CFS (check-worthy): 79% precision, 74% recall, 0.77 F-measure
• NFS (non-factual): 90% precision, 96% recall, 0.93 F-measure
• UFS (unimportant factual): 65% precision, 26% recall (most ambiguous category)

Ranking accuracy on past debates (various k values):

• Top 100 sentences: 96% precision—strong agreement with human coders
• Top 250 sentences: 94% precision

2015 Republican debate validation against professional fact-checkers:

• 6 CNN fact-checks: average ClaimBuster score 0.457 vs. 0.262 for non-checked sentences (t=3.83, p<0.001)
• 9 FactCheck.org checks: average score 0.558 vs. 0.261 (t=7.23, p<0.00001)
• 20 PolitiFact checks: average score 0.433 vs. 0.260 (t=6.67, p<0.00001)

71% of professionally fact-checked claims (27 of 38 from CNN, FactCheck.org, and PolitiFact combined) ranked in ClaimBuster's top 250 of 1,393 sentences.

Notes

Strengths:

• Clearly identifies the gap between misinformation velocity and fact-checker capacity—a key motivation for automation
• Practical tool deployed on a real debate; validation against actual fact-checker selections is credible
• Comprehensive feature engineering and feature selection analysis; cardinal numbers as the strongest signal is intuitive and replicable
• Honest about limitations: the "Holy Grail" of full automation may remain out of reach, but pursuing it drives progress on practical sub-problems

Weaknesses:

• UFS detection is weak (26% recall)—the middle category between opinions and important facts is genuinely ambiguous and may require domain or context beyond single sentences
• Limited evaluation on social media claims; training only on debate transcripts may not generalize to Twitter, Facebook, or less structured discourse
• No analysis of temporal dynamics—what makes a claim check-worthy can shift as events unfold (a claim about a future event may become check-worthy later)
• Feature engineering is labor-intensive; more recent work using pre-trained language models (BERT, RoBERTa) likely achieves better performance without hand-crafted features
• The tool identifies which sentences to check, not whether claims are true—still requires human fact-checkers for verification

Impact: This work established check-worthiness detection as a canonical research problem and provided an early benchmark. ClaimBuster has been widely cited and influenced subsequent work on automated claim identification for fact-checking pipelines.