Credibility Assessment of Textual Claims on the Web

Contributions

• Generalizes credibility assessment beyond structured subject-predicate-object triples to free-form textual claims
• Proposes joint modeling of reporting article language and web-source reliability for credibility prediction
• Introduces distant supervision approach to train classifier when labels are available per-claim, not per-article
• Demonstrates cross-domain generalization by training on Snopes and testing on Wikipedia hoaxes/fictitious people

Method

The approach operates in four stages:

Stage 1: Retrieval. For each target claim, the method uses a search engine to identify reporting articles from multiple web sources.

Stage 2: Language Feature Extraction. Each article is analyzed for stylistic features using lexicons for assertive verbs, factive verbs, hedges, implicatives, report verbs, discourse markers, and subjectivity/bias indicators. The intuition is that true claims are reported objectively, while false claims appear in sensationalized or highly subjective language.

Stage 3: Source Reliability Features. The method captures the reliability of each web source using (1) source identity as a one-hot vector, (2) PageRank (counting in/out-link quality), and (3) AlexaRank (based on unique visitors and page views). This captures the observation that sources like theonion.com (satirical) vs. WHO.org (authoritative) differ in credibility.

Stage 4: Credibility Classification. An L1-regularized logistic regression classifier learns a joint model over language and source features. Distant supervision attaches the observed claim label to all its reporting articles during training. At test time, the model produces credibility opinions for individual articles; these are aggregated by summing per-article credibility probabilities to yield the final claim verdict.

Results

On Snopes (4,856 claims, 133,272 articles):

• LG + SR model: 71.96% overall accuracy, 75.43% true-claim accuracy, 70.77% fake-claim accuracy, 0.80 AUC
• Language-only model: 69.43% accuracy
• Source-reliability-only model: 66.52% accuracy
• Baselines: FactChecker baseline (55.29%), ZeroR trivial baseline (73.69% by predicting majority class)

High precision (0.89) on fake claims shows strong performance on hoax detection. On Wikipedia hoaxes and fictitious people (trained on Snopes): 84% and 66.07% accuracy respectively, demonstrating generalization.

Notes

Strengths: The paper addresses a genuinely open-domain problem — arbitrary natural-language claims without structured assumptions. Cross-domain evaluation (Snopes → Wikipedia) is strong evidence for generalization. The combination of linguistic and source-reliability signals is intuitive and effective.

Weaknesses: Source reliability via PageRank/AlexaRank captures authority and popularity, but not credibility-specific trustworthiness. The authors note The Onion has high PageRank despite being satirical. Language features miss implicit negation ("the spokesperson denied that X occurred" about a false claim X). Error analysis shows the model struggles with well-written refutations from reputable sources. The paper does not provide interpretable evidence explaining individual predictions, only probability scores.

Future directions: Automated evidence generation to explain verdicts; refinement of negation handling; learning source credibility rankings directly from data rather than via generic popularity metrics.