Rumor has it: Identifying Misinformation in Microblogs

Vahed Qazvinian; Emily Rosengren; Dragomir R. Radev; Qiaozhu Mei

In brief

A foundational study on automated rumor detection in Twitter that proposes a framework combining content-based features (lexical, part-of-speech), network-based features (user retweet patterns), and Twitter-specific memes (hashtags, URLs) to identify rumors and classify user beliefs. The authors manually annotate 10,000 tweets across five rumors and achieve 96.5 MAP for rumor retrieval and 93.5% accuracy for belief classification, demonstrating that combined feature sets substantially outperform content-only approaches.

Summary figure for Rumor has it: Identifying Misinformation in Microblogs

Contributions

First large-scale manually annotated dataset of Twitter rumors (~10,400 tweets across 5 rumors: obama, airfrancue, cellphone, michelle, palin)
Framework for two-stage rumor identification: (1) rumor retrieval (retrieve all tweets discussing a specific rumor), (2) belief classification (identify whether users endorse, refute, or question the rumor)
Comprehensive feature engineering combining content, network, and Twitter-specific signals
Statistical models (L1-regularized log-linear models) achieving strong performance on both tasks

Method

The approach frames rumor identification as two related NLP tasks:

Rumor Retrieval: Given seed rumors, retrieve all relevant tweets from a corpus. Uses L1-regularized log-linear models combining log-likelihood ratios of features computed from positive and negative training sets. Features include:

Content-based: lexical unigrams/bigrams, part-of-speech unigrams/bigrams
Network-based: whether the tweeting user has posted/retweeted rumors before (user model), whether a tweet is retweeted by users with a history of spreading rumors (re-tweet user model)
Twitter-specific: hashtags and URLs extracted from tweets. URLs are fetched and their content modeled with unigram/bigram language models

Belief Classification: Among tweets flagged as rumorous, identify whether the poster believes the rumor, refutes it, questions it, or is neutral. Uses the same feature sets (content, network, Twitter signals) to train classifiers on labeled tweets; a tweet is marked as "believe" if the poster endorses the rumor statement, "doubt" if they explicitly deny or question it, or "neutral" if they express no clear stance.

The optimization employs orthogonal-wise limited-memory quasi-Newton (OWL-QN) for efficient L1-regularized objective minimization.

Results

Rumor Retrieval (Information Retrieval framework):

L1-regularized log-linear model with all 9 features: Mean Average Precision (MAP) 0.965 (95% CI: [0.936, 0.994])
Substantially outperforms regex-only baseline (MAP 0.587) and KL-divergence baseline (MAP 0.803)
Content-based features (precision/recall tradeoff) + network features show strongest gains

Belief Classification:

All-feature model: 93.5% accuracy, 94.4% precision, 90.6% recall, F_{β=1} = 0.925
Content features alone (TXT+POS): 94.1% accuracy
Network features (USR): 84.8% accuracy
Twitter-specific (TAG+URL): 68.3% accuracy
Combining all three feature types provides significant improvement over any single set
Win/loss ratio (correct/incorrect): 14.395 (indicating strong discriminative power)

Notes

This is one of the first large-scale computational studies of rumors on social media. The manual annotation of >10,000 tweets at high inter-rater agreement (κ=0.95 for rumor retrieval, κ=0.85 for belief classification) establishes a rigorous standard for the field. The three-category feature framework (content, network, Twitter-specific) became a de facto standard for early social-media misinformation detection. Key insight: network-based features (user history of rumor involvement) and Twitter-specific signals (hashtags, URLs) substantially improve performance over content alone, foreshadowing the importance of social-graph and platform-specific signals in later work. The paper's distinction between rumor retrieval (finding all discourse about a claim) and belief classification (determining user stance) cleanly separates two related but distinct problems.