Contributions
- Annotation scheme for rumorous social media conversations with three dimensions: support/response type, certainty, and evidentiality. Achieves inter-annotator agreement of 62%+ on individual dimensions.
- PHEME dataset: 330 annotated rumorous conversations (4,842 tweets) from 9 newsworthy events (Sydney siege, Ferguson unrest, Charlie Hebdo shooting, Germanwings crash, Ottawa shooting, Putin missing, Prince rumor, Essien/Ebola, Gurlitt collection).
- Analysis showing users with high follower ratios (typically news organizations) support rumors with high certainty and provide external evidence, while low-reputation users tend to deny rumors.
Method
The authors collected tweets related to 9 newsworthy events likely to spawn rumors. They used Twitter's streaming API to capture tweets containing rumor-related keywords, then manually identified rumorous source tweets with the help of journalists. Conversational threads were reconstructed by retrieving all replies and nested interactions to source tweets.
The annotation scheme captures three independent dimensions for each tweet in a rumorous conversation:
- Support and Response Type — whether a source tweet supports, denies, or leaves unspecified the rumor; whether responses agree, disagree, appeal for more information, or comment.
- Certainty — the author's confidence level (certain, somewhat certain, or uncertain) when posting.
- Evidentiality — types of evidence provided: first-hand experience, URL attachment, quote/attribution, picture, reasoning, or none.
Rumorous stories were grouped and annotated for ultimate veracity (true/false/unverified), with a "resolving tweet" marked when available to distinguish before/after resolution.
Results
Out of 2,695 annotated rumorous tweets, 250 rumorous stories were identified: 159 remaining unverified, 68 proven false, 103 ultimately true.
Resolution timelines: True rumors resolve significantly faster (median 2 hours, most within 5 hours) than false rumors (median 14 hours, many not resolved until 15-20 hours post-source tweet). This aligns with intuition that disproving a claim is harder than proving one true.
Rumor spread patterns: Unverified rumors generate a distinctive burst of retweets in the first 15 minutes. Early tweets supporting unverified rumors spark 3.7× more retweets than early debunking tweets. Once a resolving tweet is posted, retweets decline for both true and false rumors.
Support ratios: Before resolution, true rumors have median support ratio of 0.156 vs. false rumors 0.067 — both slightly favoring support. After resolution, support drops for both (true: median 0.021, false: median −0.072), suggesting users stop promoting stories once veracity is settled.
User reputation: Users with high follower ratios (log ratio ≥ 4) are predominantly news organizations and show significantly higher support ratios and higher use of external evidence when engaging with rumors, both before and after resolution.
Certainty and evidence: Unverified rumors show lower certainty (median 0.571) similar to eventually-true rumors (0.575). False rumors show lower certainty than true ones before resolution (0.618 vs. 0.575) but the difference is not statistically significant. After false rumors are debunked, certainty drops (median 0.556), consistent with users shifting focus from supporting to debunking claims.
Notes
This is a landmark dataset paper introducing conversational-thread analysis to the rumor detection field. The annotation scheme's three independent dimensions allow fine-grained analysis of user behavior. The key finding that users preferentially share unverified claims before resolution has implications for intervention design. The paper is careful to distinguish support orientation from rumor truth value, revealing that certainty and evidence alone do not drive effective rumor debunking.