Skip to content

In brief

ClaimBuster is a fully-integrated end-to-end fact-checking system that monitors live political discourse (debates, speeches, social media) in real time, scores sentences for check-worthiness, matches them against curated fact-check repositories, and automatically queries external knowledge bases and the web to generate verification reports. Deployed during the 2016 U.S. presidential election and Australian parliament proceedings, the system's check-worthiness detector achieves 74% recall and 79% precision and shows high agreement with professional fact-checkers at CNN, PolitiFact, and FactCheck.org.

Summary figure for ClaimBuster: The First-ever End-to-end Fact-checking System

Contributions

  • An integrated automated fact-checking pipeline with five main components: claim monitor, claim spotter (check-worthiness detector), claim matcher, claim checker, and fact-check reporter.
  • A claim spotter model trained on tens of thousands of manually-labeled sentences from election debates, achieving 74% recall and 79% precision in detecting check-worthy factual claims.
  • Real-world deployment across multiple data sources: live TV closed captions from 25 primary and general election debates in 2016, Twitter streams from 2,220 political accounts, and Australian parliament (Hansard) transcripts.
  • Evaluation showing strong correlation ($r > 0.8$) between ClaimBuster's claim rankings and decisions by professional fact-checkers across three major organizations.
  • Multi-channel fact-check reporting infrastructure: web interface, Twitter feed (@ClaimBusterTM, 13K+ claims posted), Slack bot, and public API.

Method

Claim Monitor continuously ingests text from multiple sources:

  • Broadcast media: closed captions extracted from live TV via decoding devices; speaker identification derived from concurrent Twitter Streaming API activity during major events
  • Social media: filters 2,220 curated Twitter accounts (politicians, news organizations) using SVM-based classifier to separate politics-related content
  • Websites: monitors archived proceedings (e.g., Hansard parliamentary transcripts)

Claim Spotter scores each sentence on a continuous scale [0, 1] indicating check-worthiness. The classification model is trained on tens of thousands of manually-coded sentences from general election debates, using features including token n-grams and part-of-speech tags. The model learns to distinguish factual, check-worthy claims (high score) from subjective, opinionated statements (low score).

Claim Matcher searches a curated fact-check repository (aggregated from PolitiFact, CNN, FactCheck.org and other professional sources) using two similarity metrics: token-based similarity (Elasticsearch) and semantic similarity (Semilar toolkit). Results from both methods are combined to identify matching or closely-related fact-checks.

Claim Checker generates verification evidence when no repository match is found, using two approaches:

  1. Question generation: transforms the claim into questions, sends them to Wolfram Alpha and Google, and compares returned answers against the original claim to detect discrepancies
  2. Web search: issues the claim as a search query to Google, downloads and parses top results, extracts matching sentences and surrounding context

Fact-check Reporter synthesizes evidence from both matching and checking processes and delivers reports through multiple channels: the web interface (showing matched fact-checks with verdicts, QA results, and search context), Twitter feed, Slack bot, and public REST API.

Results

Claim spotter evaluation: 74% recall and 79% precision on manually-coded test sentences from 2016 presidential debates.

Correlation with professional fact-checkers: Post-hoc analysis of claims identified during primary debates reveals a highly positive correlation (Pearson $r > 0.80$, $p < 0.001$) between ClaimBuster's check-worthiness rankings and the decisions of professional fact-checkers at CNN, PolitiFact.com, and FactCheck.org regarding which claims to fact-check [5]. The model also generalizes well to politics-related Twitter content and Australian parliamentary proceedings, despite being trained exclusively on debate data.

Deployment scale: System has been monitoring Twitter continuously since deployment, accumulating 13,000+ check-worthy claims posted to @ClaimBusterTM. Successfully processed all 25 debates (21 primary, 4 general) in the 2016 U.S. election cycle with live ranking available in real time.

Notes

This demonstration paper introduces the first fully-integrated end-to-end system architecture for automated fact-checking, moving beyond individual component research. The key strength is the real-time deployment across multiple data sources (debates, Twitter, Hansard) with live human-expert validation showing high agreement. The check-worthiness component is now standard in the field. Limitations include reliance on curated repositories for coverage and difficulty generating verdicts for novel claims without matching fact-checks. The question-generation and evidence-synthesis approaches are hand-engineered; modern work uses learned ranking and generation. The 2016 election deployment was historically significant as one of the first large-scale automated fact-checking applications in real political discourse.

Similar papers

Discussed in GUIDE by

Summarized from the full text. Added 2026-11-17 from a submission by reza. Every statement on this page traces back to the source.