Real-time Rumor Debunking on Twitter

Contributions

• First real-time rumor debunking algorithm for Twitter, operating at event level rather than individual tweets.
• Dataset of 842 events (421 true, 421 false stories) collected from Snopes and Emergent, with associated tweets gathered via keyword queries and Twitter streaming.
• Feature hypotheses rooted in investigative journalism: source credibility, source diversity, witness accounts, belief partitions (support, denial, question, neutral).
• Demonstration that early predictions (5 tweets, 1 hour) outperform baseline methods and achieve 75%+ speed advantage over human verification.

Method

The system models rumor debunking as supervised binary classification of events (truth vs. rumor). At time t, given a set of tweets P(t) associated with an event, a classifier M(t) predicts veracity using aggregated features across all tweets in the event.

Event detection: Clustering algorithm groups tweets by keyword queries formulated as A ∧ B (conjunction of entities AND disjunction of action words; e.g., "macklemore AND (joins OR participates) ISIS").

Feature categories (720 total after aggregation):

1. Source verification (inspired by journalism): trusted/satirical news account, presence of trusted/satirical URLs, domain reputation, application type.
2. Source identity: profile contains person name, location, profession information, multiple URLs.
3. Source diversity: fraction of tweets with distinct news/non-news URLs, semantic dissimilarity of deduplicated tweet text.
4. Witness features: presence of witness phrases ("I see", "I heard") indicating first-hand observation.
5. Belief identification: classify users into support, denial, question, or neutral stances toward the event using rule-based keywords (e.g., "negative words not preceded by negation imply denial"). Aggregate belief features by partitioning event tweets into belief groups.
6. Propagation: retweet counts, mention graph structure, hashtag h-index, reply/retweet depth and breadth.

Numeric features aggregated as min/max/average; Boolean features converted to fractions; categorical features decomposed into Boolean arrays. Hypotheses: (H1) verification features improve detection; (H2) belief partitions improve detection.

Classification: 10-fold cross-validation with SVM (outperformed random forest and decision tree). Best-first feature selection.

Results

Static accuracy (10-fold CV):

• 5-tweet baseline (Castillo et al.): 65.5%; +H1: 73.6%; +H1+H2: 78.0%
• 100-tweet: Castillo 72.9%, +H1+H2: 85.7%
• 400-tweet: Castillo 74.5%, +H1+H2: 89.7%

Timeliness vs. human verification:

• 75% of rumors debunked by the system before Snopes/Emergent verification, at 85%+ accuracy.
• 75% of rumors debunked before news media, at 80%+ accuracy (measured on 86 manually verified rumors).

Feature importance: User characteristics dominate consistently. Belief features comprise 21% of top features in early prediction (5 tweets) but rise to 43% at 400 tweets, showing increasing importance as more information accumulates.

Notes

This is an early system paper applying supervised learning to rumor debunking on social media. Strengths: real-time prediction setting is practical and motivated; belief partitions are an underexplored signal; the wisdom-of-crowds framing is intuitive. Limitations: event detection is keyword-based and may miss related stories with different terminology; Twitter's API limitations at the time prevented tracking retweet cascades in detail, making comparison with Sina Weibo studies difficult; no discussion of deployment or scalability; class balance (421 true, 421 false) is artificial (real Twitter is imbalanced); later work on stance detection and NLP-based belief identification would refine the belief classification approach. The paper does not address how to handle early predictions with very few tweets and how confidence should degrade; deployment robustness is not discussed.