Contributions
- Comprehensive measurement dataset: Assembled fake news sites from five prior lists (academic papers, PolitiFact, BuzzFeed, FactCheck); measured BuzzSumo data on Facebook engagements and Twitter shares at monthly granularity over 3.5 years.
- Platform divergence signal: First to document sharp divergence between Facebook and Twitter trends for false content following the 2016 election, providing indirect evidence that platform interventions may have worked.
- Robustness against confounds: Showed that non-fake-news sites (major news, small news, business/culture) followed parallel trends on both platforms before and after the election, suggesting the divergence is specific to misinformation.
- URL-level analysis: Complemented site-level analysis with a filtered set of 9,540 false story URLs extracted from Snopes, showing similar patterns.
Method
Data sources:
- 569 fake news sites identified from union of five lists: Grinberg et al. (2019), PolitiFact, BuzzFeed articles, Guess et al. (2018), FactCheck—plus smaller sites from Snopes, Zimdars (2016), and Brayton (2016).
- Comparison groups: 38 major news sites (top 100 by Alexa), 78 small news sites (rank 401–500), 54 business/culture sites (top 50 in Arts, Business, Health, Recreation, Sports categories).
- Engagement metrics from BuzzSumo (tracks user interactions via Facebook API and Twitter purchase data): monthly Facebook engagements (shares + comments + reactions) and Twitter shares for all URLs, January 2015 through July 2018.
- Supplemental URL-level analysis: 9,540 false story URLs scraped from Snopes (flagged "false" or "mostly false"), filtered to articles with ≥100 Facebook engagements or ≥10 Twitter shares, manually screened.
Analysis: Quarterly aggregation of monthly totals; trend comparison of fake news vs. comparison groups; focus on ratio of Facebook engagements to Twitter shares as a robustness check against sample selection and demand changes.
Results
Site-level trends:
- Fake news engagements rose from ~60M/month (Q1 2015) to ~160M/month (Q4 2016) on Facebook; fell to ~60M/month by Q3 2018.
- Twitter shares of fake content rose from ~1.5M/month to ~4–5M/month over the same period and continued rising.
- Non-fake-news sites (major, small, business/culture) remained stable or showed modest trends, with consistent Facebook/Twitter ratios (~10:1 to ~15:1).
Key metric—Facebook-to-Twitter ratio for fake news:
- ~45:1 from 2015 through late 2016 (steady).
- Fell sharply to ~15:1 by end of sample (July 2018).
- Same stability observed for non-fake-news comparison groups.
URL-level analysis (false stories):
- Similar pattern: Facebook engagement/Twitter share ratio declined by half or more after the 2016 election.
- Median ratio of Facebook engagements to Facebook shares: ~6:1 for false URLs.
Robustness checks: Findings held across multiple robustness scenarios: restricting to sites identified by 2+ or 3+ lists, excluding sites identified by only one list, restricting to recently launched sites, excluding top 5 sites, separate analyses for large vs. small sites, alternative site definitions from Grinberg et al. (2019), and using Facebook shares instead of total engagements.
Notes
Strengths:
- Novel longitudinal data spanning three critical years (2015–2018), including the pivotal 2016 election.
- Large, systematically constructed dataset of fake news sites from multiple independent sources, reducing single-list bias.
- Clever use of platform comparison (Facebook vs. Twitter ratio) to partially isolate platform-specific effects from other confounds (changing demand, domain evasion).
- Extensive robustness checks demonstrate findings are not driven by outliers or sample selection artifacts.
Limitations:
- BuzzSumo data reflects user interactions, not exposure or belief; engagement ≠ impact.
- Results measure relative trends (Facebook/Twitter ratio) more reliably than absolute decline; absolute numbers subject to selection bias in site identification and measurement gaps.
- Cannot definitively attribute decline to Facebook's interventions; other factors (media literacy efforts, demand shifts, evasion by producers) could contribute.
- 103 fake news sites lack BuzzSumo coverage (likely small sites); impact on totals is small but not ruled out.
- Snopes-based URL filtering may over-represent certain types of false claims.
Significance: First large-scale, longitudinal measurement of platform-level differences in misinformation spread. Provides rare direct evidence (via platform comparison) that platform interventions may reduce misinformation relative to organic demand. Influential in policy and platform accountability discussions post-2016.