Skip to content
Detection methodsFact-checking & verification check worthy claimsrankingfeature engineering

A Context-Aware Approach for Detecting Check-Worthy Claims in Political Debates

Recent Advances in Natural Language Processing (RANLP) · 2017

In brief

Identifying which claims in political debates warrant fact-checking is crucial but understudied. Rather than isolating sentences, this work models context—the relationship between a claim and the broader debate, opponent interactions, and audience reactions. The authors create a dataset from 2016 US debates fact-checked by nine reputable sources and train both SVM and neural network models with rich contextual features, achieving state-of-the-art ranking performance and demonstrating that context substantially improves check-worthiness prediction.

Summary figure for A Context-Aware Approach for Detecting Check-Worthy Claims in Political Debates

Contributions

  • New dataset (CW-USPD-2016): Manually annotated claims from four 2016 US presidential and vice-presidential debates, fact-checked by nine reputable sources (CNN, NPR, PolitiFact, FactCheck.org, NYT, Washington Post, The Guardian, Chicago Tribune, ABC News). Dataset contains 880 annotated sentences across 5,415 total sentences.
  • Context-aware feature engineering: Novel approach modeling not just the target sentence but its contextual relationships: position in the speaker's segment, segment sizes, opponent interactions, moderator and audience reactions, topic distributions, embeddings, discourse relations, and contradiction patterns.
  • State-of-the-art ranking results: Both SVM and feed-forward neural networks outperform the ClaimBuster baseline. FNN achieves MAP of .427 (vs .317 for ClaimBuster) and demonstrates that contextual features provide substantial performance gains.

Method

The authors frame check-worthiness prediction as a ranking task. They develop a comprehensive feature set operating at sentence (S), context (C), or both levels:

Sentence-level features: ClaimBuster reimplementation (TF-IDF bag-of-words, POS tags, named entities, sentiment from Alchemy API, length in tokens); NRC sentiment lexicon features; named entity counts; linguistic features from established lexicons (bias, negation, factives, assertives, hedges, implicatives, subjective words); tense (past/present/future); and sentence length in characters.

Contextual features: Position within speaker segment (first/last, reciprocal rank); previous/current/next segment sizes; metadata (opponent mention, speaker identity, audience reactions—applause, laughter, cross-talk).

Mixed sentence+context features: Topic modeling (300-topic LDA trained on presidential speeches 2007–2016); 300-dimensional word embeddings (Google News, with contextual similarity); discourse parsing (nuclei/satellites, discourse relations like CONTRAST, CAUSE); negation-based contradiction detection; k-NN similarity to training examples.

Models trained with SVM (RBF kernel) and feed-forward neural networks (2 hidden layers: 200, 50 neurons, ReLU, softmax output). Training objective: binary classification (check-worthy if one or more media flagged it), then rank by classifier scores. Evaluation: 4-fold cross-validation, one debate per fold, using ranking metrics (MAP, R-Precision, Precision@k for k ∈ {5,10,20,50}).

Results

Main results (FNN with all features): MAP .427, R-Pr .432, P@5 .800, P@10 .725, P@20 .713, P@50 .600.

Comparison: Outperforms all three ClaimBuster variants (online platform MAP .317, SVM reimplementation .360, FNN reimplementation .357) and TF-IDF baseline (.314).

Feature ablation: Embeddings alone yield MAP .357; removing contextual features drops FNN performance from .427 to .385 (MAP); contextual features only score .317 (comparable to ClaimBuster).

Per-source adaptation: System generalizes across media sources; training on all media performs better than source-specific retraining for most outlets, suggesting shared check-worthiness principles.

Notes

Strengths:
The dataset's multi-source annotation from real fact-checking organizations is a major asset—it captures genuine editorial judgment rather than synthetic labels. The contextual feature engineering is thorough and well-motivated; the finding that context provides a large performance lift (MAP delta of .042 when removed) is convincing. The paper's transparency about inter-annotator agreement (only 1 sentence agreed by all 9 sources; 492 by exactly 1) is honest about the task's difficulty.

Limitations:
Agreement between sources is very low, suggesting the task may be partly subjective; the paper does not deeply explore why different media select different claims. The dataset is restricted to 2016 US debates and may not generalize to other debate types, languages, or time periods. Some false positives identified in error analysis are genuinely borderline cases—the system's ranking behavior may be nearly optimal given the noisy labels.

Follow-up work: Extending the dataset to other elections, interviews, and general political discourse; moving beyond sentence-level claims to sub-sentence fragments; coupling with automatic fact-checking systems to create end-to-end pipelines.

How it connects

Similar papers

Discussed in GUIDE by

Summarized from the full text. Added 2026-11-16 from a submission by reza. Every statement on this page traces back to the source.