Contributions
- Dataset of 1,571 labeled sentences from U.S. presidential debate transcripts (2004, 2008, 2012), categorized as non-factual, unimportant factual, or check-worthy factual claims.
- Comparison of three multi-class classification approaches (Naive Bayes, SVM, Random Forest) with multiple feature sets.
- Analysis of feature importance, showing that numeric content (POS tag CD) is the strongest discriminator between factual claim types.
- Demonstration that check-worthiness can be automatically detected with reasonable precision and recall, enabling fact-checking prioritization systems.
Method
The paper frames claim prioritization as a three-way text classification task. Sentences are categorized as:
- Non-factual sentences (NFS): opinions, beliefs, declarations, and most questions
- Unimportant factual sentences (UFS): true claims the public has no interest in verifying (e.g., "Next Tuesday is Election Day")
- Check-worthy factual sentences (CFS): claims about important topics that merit verification (e.g., "He voted against the first Gulf War")
Features extracted include sentiment scores (via AlchemyAPI), sentence length, tf-idf word features, POS tag counts (43 tags), and entity types (26 types). After feature selection via random forest GINI importance, the authors trained three classifiers: Multinomial Naive Bayes, Support Vector Machine, and Random Forest, all using 4-fold cross-validation on feature combinations ranging from single categories to the 100 most important features.
Results
SVM with Word + POS Tag features achieved the best weighted average performance: 70% precision, 72% recall, 70% F-measure. Random Forest and SVM outperformed Naive Bayes. The numeric POS tag (CD) emerged as the single most discriminative feature—45% of check-worthy claims contained numbers, versus only 6% of non-factual sentences. Inter-annotator agreement using Cohen's kappa showed RFC and SVM achieved moderate agreement (κ ≈ 0.45–0.48) with human judges, versus fair agreement (κ ≈ 0.01–0.40) for Naive Bayes.
Notes
This is an early and often-cited contribution to computational fact-checking, establishing the practical problem of claim prioritization. The dataset, though small by modern standards (1,571 sentences), was influential. A key limitation is the class imbalance: only 252 (16%) of labeled sentences are unimportant factual claims, making UFS harder to classify than NFS or CFS. The approach is purely syntax-driven; semantic understanding of claim content is not used. The model was designed with 2016 election deployment in mind but no production deployment was documented.