Rumor Detection over Varying Time Windows

Sejeong Kwon; Meeyoung Cha; Kyomin Jung

doi:10.1371/journal.pone.0168344

In brief

This paper examines how rumor classification performance changes across different observation windows (3, 7, 14, 28, and 56 days) using user, linguistic, network, and temporal features extracted from Twitter. The key finding is that user and linguistic features are predictive during early circulation but their importance decreases over time, while network and temporal features become increasingly predictive in longer windows. The authors propose two new classification algorithms optimized for different time windows.

Summary figure for Rumor Detection over Varying Time Windows

0.76Highly predictive in 3-day window (F1: 0.76)

0.80Remarkably stable predictors (F1: 0.80–0.86 across all windows)

0.77Dramatically improve with longer observation (56 days: F1 = 0.77)

0.70Become significant at 14+ days (F1 0.70)

Contributions

Temporal analysis of feature importance: Demonstrates that rumor characteristics evolve over time, with different feature sets becoming salient at different propagation stages
Comprehensive feature framework: Examines 90+ features across four domains (user, linguistic, network, temporal) grounded in social psychology and information diffusion theory
Periodic External Shock (PES) model: Proposes a temporal model capturing cyclical patterns in rumor spreading by extending the SIkεM epidemic model
Practical algorithms: Introduces two rumor classification algorithms—one for early detection (user + linguistic features) and one for long-term analysis (all features)—each with different computational and data requirements
Large-scale empirical validation: Uses 111 events (60 rumors, 51 non-rumors) from Twitter's near-complete archive covering 2006–2009

Method

Data

The authors extracted rumor and non-rumor events from Twitter's historical data:

Rumors: Two rumor archives (snopes.com, urbanlegends.about.com) were used to identify rumorous events and their occurrence dates on Twitter
Non-rumors: News events from times.com, nytimes.com, cnn.com were used to identify notable non-rumorous events
Dataset: 111 total events (60 rumors, 51 non-rumors) with high inter-annotator agreement (ICC > 0.99)
Time scope: 2006–2009 tweets and network information

Features

User features (27 variables): Followers, friends, tweets, percentiles (25th, 50th, 75th, 100th) of these counts, means, standard deviations, skewness, kurtosis. Hypothesis: rumor spreaders have distinctive audience sizes and engagement patterns.

Linguistic features (21 variables): Word counts from the Linguistic Inquiry and Word Count (LIWC) tool capturing cognitive styles (skepticism, negation), emotional dimensions (tentative, certain), and social processes (hearing, family, social relationships). Theory: rumors trigger different linguistic patterns than factual news.

Network features (43 variables): Three network types—

Extended network (all users touching an event)
Friendship network (followers/following relationships)
Diffusion network (inferred information flow based on posting order and follow relationships)

Measures: density, clustering coefficient, degree distribution, connected component size, and direction of information flow. Theory: rumors create denser, smaller communities with weaker external connections than news.

Temporal features (5 variables): Periodic External Shock (PES) model parameters capturing periodicity, amplitude, and phase of rumor cycles, plus a baseline shock strength.

Classification

Machine learning: Random forest classifiers with 10-fold cross-validation
Feature selection: Ran stepwise variable selection 10 times, selecting features that appeared ≥8 times to ensure robustness
Comparison: Evaluated each feature domain independently and in combination across the five observation windows

Results

Feature importance by observation window

User features:

Highly predictive in 3-day window (F1: 0.76)
Decline over time; by 56 days, F1 drops to 0.64
Interpretation: early rumor spreaders have distinctive profiles, but these characteristics converge as more diverse users engage

Linguistic features:

Remarkably stable predictors (F1: 0.80–0.86 across all windows)
Key indicators: skepticism, negation, tentative language, and words related to cognition and social relationships
Most useful alone for early detection; combined with others, they maintain predictive power
F1 score of 0.80 using linguistic features alone; 0.8–0.86 when combined with others

Network features:

Poor performance in short windows (3–7 days: F1 ≈ 0.56–0.65)
Dramatically improve with longer observation (56 days: F1 = 0.77)
Interpretation: rumor diffusion networks take time to develop distinctive structure; early network is too sparse

Temporal features:

No predictive power in short windows (3–7 days)
Become significant at 14+ days (F1 > 0.70)
Most predictive at 56 days (F1 = 0.87) when periodic cycles are apparent
Encoding: rumor events show repeating peaks; non-rumor events show single prominent spike

Overall classification performance

Across all features combined:

3 days: 0.74 accuracy
7 days: 0.73 accuracy
14 days: 0.75 accuracy
28 days: 0.70 accuracy
56 days: 0.89 accuracy (F1 = 0.90)

Proposed algorithms

Algorithm 1 (early rumor detection): Uses user and linguistic features only

F1 score: 0.77–0.80 at 28 days (best: 0.80)
Advantage: detectable without full network history; practical for real-time systems

Algorithm 2 (long-term classification): Uses all features

F1 score: 0.80–0.90 (best: 0.90 at 56 days)
More accurate but requires complete network data and extended observation periods

Notes

Strengths:

Comprehensive feature analysis with strong theoretical grounding in psychology and network science
Large-scale study using near-complete Twitter archive; findings are statistically robust
Practical insight: early detection requires different tools than long-term analysis—informs system design
Reproducible methodology; authors describe feature extraction and model fitting in detail
Stable linguistic features offer practical advantage for real-time systems with incomplete network data

Limitations:

Limited to Twitter; generalization to other platforms or languages unclear
Rumor/non-rumor ground truth relies on manual archives (snopes.com, urbanlegends); some events may be misclassified or miss borderline cases
Feature sets derived from common case studies; may not generalize to novel types of rumors
Network completeness assumption: the authors use the public Twitter API snapshot from 2009, which may be incomplete for users who deleted tweets or accounts
Machine learning features are observational; causation between spreading patterns and veracity cannot be claimed
Limited to single language (English) and cultural context

Impact and follow-up:

This paper established temporal variation in rumor detection features as a core research direction. Subsequent work has extended this to: real-time early detection (building on the 3–7 day insights), multimodal rumor detection, and cross-platform rumor tracking. The finding that linguistic features remain stable predictors has been validated in later work on fact-checking and claim verification.