
Contributions
- An integrated automated fact-checking pipeline with five main components: claim monitor, claim spotter (check-worthiness detector), claim matcher, claim checker, and fact-check reporter.
- A claim spotter model trained on tens of thousands of manually-labeled sentences from election debates, achieving 74% recall and 79% precision in detecting check-worthy factual claims.
- Real-world deployment across multiple data sources: live TV closed captions from 25 primary and general election debates in 2016, Twitter streams from 2,220 political accounts, and Australian parliament (Hansard) transcripts.
- Evaluation showing strong correlation ($r > 0.8$) between ClaimBuster's claim rankings and decisions by professional fact-checkers across three major organizations.
- Multi-channel fact-check reporting infrastructure: web interface, Twitter feed (@ClaimBusterTM, 13K+ claims posted), Slack bot, and public API.
Method
Claim Monitor continuously ingests text from multiple sources:
- Broadcast media: closed captions extracted from live TV via decoding devices; speaker identification derived from concurrent Twitter Streaming API activity during major events
- Social media: filters 2,220 curated Twitter accounts (politicians, news organizations) using SVM-based classifier to separate politics-related content
- Websites: monitors archived proceedings (e.g., Hansard parliamentary transcripts)
Claim Spotter scores each sentence on a continuous scale [0, 1] indicating check-worthiness. The classification model is trained on tens of thousands of manually-coded sentences from general election debates, using features including token n-grams and part-of-speech tags. The model learns to distinguish factual, check-worthy claims (high score) from subjective, opinionated statements (low score).
Claim Matcher searches a curated fact-check repository (aggregated from PolitiFact, CNN, FactCheck.org and other professional sources) using two similarity metrics: token-based similarity (Elasticsearch) and semantic similarity (Semilar toolkit). Results from both methods are combined to identify matching or closely-related fact-checks.
Claim Checker generates verification evidence when no repository match is found, using two approaches:
- Question generation: transforms the claim into questions, sends them to Wolfram Alpha and Google, and compares returned answers against the original claim to detect discrepancies
- Web search: issues the claim as a search query to Google, downloads and parses top results, extracts matching sentences and surrounding context
Fact-check Reporter synthesizes evidence from both matching and checking processes and delivers reports through multiple channels: the web interface (showing matched fact-checks with verdicts, QA results, and search context), Twitter feed, Slack bot, and public REST API.
Results
Claim spotter evaluation: 74% recall and 79% precision on manually-coded test sentences from 2016 presidential debates.
Correlation with professional fact-checkers: Post-hoc analysis of claims identified during primary debates reveals a highly positive correlation (Pearson $r > 0.80$, $p < 0.001$) between ClaimBuster's check-worthiness rankings and the decisions of professional fact-checkers at CNN, PolitiFact.com, and FactCheck.org regarding which claims to fact-check [5]. The model also generalizes well to politics-related Twitter content and Australian parliamentary proceedings, despite being trained exclusively on debate data.
Deployment scale: System has been monitoring Twitter continuously since deployment, accumulating 13,000+ check-worthy claims posted to @ClaimBusterTM. Successfully processed all 25 debates (21 primary, 4 general) in the 2016 U.S. election cycle with live ranking available in real time.
Notes
This demonstration paper introduces the first fully-integrated end-to-end system architecture for automated fact-checking, moving beyond individual component research. The key strength is the real-time deployment across multiple data sources (debates, Twitter, Hansard) with live human-expert validation showing high agreement. The check-worthiness component is now standard in the field. Limitations include reliance on curated repositories for coverage and difficulty generating verdicts for novel claims without matching fact-checks. The question-generation and evidence-synthesis approaches are hand-engineered; modern work uses learned ranking and generation. The 2016 election deployment was historically significant as one of the first large-scale automated fact-checking applications in real political discourse.