Skip to content
A digital media literacy intervention increases discernment between mainstream and false news in the United States and India

A digital media literacy intervention increases discernment between mainstream and false news in the United States and India

Authors: Andrew M. Guess, Michael Lerner, Benjamin Lyons, Jacob M. Montgomery, Brendan Nyhan, Jason Reifler, Neelanjan Sircar

Venue: PNAS, Vol. 117, No. 27, July 7, 2020 — DOI

TL;DR

A randomized experiment tests Facebook's "Tips to Spot False News" intervention in the US and India, finding it increases discernment between mainstream and false news (26.5% in US, 17.3% in India online). The intervention reduces belief in false headlines more than it reduces belief in mainstream ones. Effects persist for several weeks in the US but decay over time, and do not materialize in a face-to-face sample of less-educated, less-digitally-connected rural Indians.

Contributions

  • First large-scale randomized controlled trial of a real-world digital media literacy intervention deployed globally
  • Demonstrates that simple, scalable "tips" interventions can improve headline accuracy discrimination across populations
  • Documents heterogeneous effects across countries and study designs (online vs. face-to-face)
  • Shows effects are not merely increasing overall skepticism but specifically improve discernment between news types
  • Provides evidence that digital media literacy deficits are a factor in susceptibility to misinformation

Method

The study uses preregistered two-wave panel surveys in three samples: a nationally representative US online sample (N=4,907), an India online convenience sample (N=3,273), and a representative face-to-face sample in rural Uttar Pradesh (N=3,744).

Intervention: Respondents randomly assigned to treatment received 10 tips (US) or 6 tips (India) adapted from Facebook's 2017 "Tips to Spot False News" campaign. Tips include advice like "be skeptical of headlines" and checking sources. Respondents answer comprehension questions after each group of tips.

Outcome measures: Respondents rate 12–16 headlines on a 4-point accuracy scale (1=not at all accurate, 4=very accurate). Headlines vary in type (mainstream, false, hyperpartisan), valence (partisan congeniality), and source prominence. The key outcomes are: perceived accuracy of false news, perceived accuracy of mainstream news, and discernment (the difference between the two).

Analysis: OLS regression with headline fixed effects, clustered standard errors by respondent. Intent-to-treat (ITT) estimates reported for the full sample; average-treatment-on-treated (ATT) estimates account for varying compliance rates across samples (66% in US, 24–28% in India).

Results

US sample (wave 1): - False news accuracy decreased by 0.196 points on 4-point scale (ITT) or 0.299 (ATT) - Mainstream news accuracy decreased by 0.046 points (ITT) or 0.071 (ATT) - Discernment improved by 0.146 points (ITT) or 0.223 (ATT), a 26.5% relative improvement - Effect persisted at wave 2 (~3 weeks later) but at reduced magnitude (false news effect: 0.080 ITT)

India online sample (wave 1): - False news accuracy decreased by 0.126 points (ITT) or 0.470 (ATT) - Mainstream news accuracy decreased by 0.071 points (ITT) or 0.259 (ATT) - Discernment improved by 0.063 points (ITT) or 0.221 (ATT), a 17.3% relative improvement - No detectable effects at wave 2 (~3 weeks later)

India face-to-face sample: - No significant effects on false news accuracy, mainstream news accuracy, or discernment - This sample had much lower social media use (11% WhatsApp vs. 90% in online sample)

Cross-study pattern: The US and India online results converge on the same pattern (false news accuracy reduced more than mainstream). A pooled analysis across all three samples confirms the overall effect is significant but heterogeneous by study design.

Connections

Notes

Strengths: - Preregistered design in both countries with publicly posted analysis plans - Large, well-powered samples with demographic diversity in the US and face-to-face arm - Real-world intervention deployed by Facebook globally, maximizing external validity - Rigorous experimental design with both ITT and ATT estimates accounting for compliance - Two-wave panel design tests durability of effects - Clear improvement in discernment (not just increased skepticism toward all news)

Limitations: - Effect sizes are modest; the intervention does not eliminate false news belief - Effects decay significantly over time (especially in India) - Small trade-off: mainstream news accuracy decreases alongside false news, though the magnitude is substantially smaller - Face-to-face results in rural India show this approach may not work for populations with low digital experience - Limited evidence that intervention changes real-world news consumption or sharing behavior - Many real-world Facebook users likely ignored the tips, limiting practical reach

Substantive findings: The results suggest that shortfalls in digital media literacy are indeed a measurable factor in susceptibility to misinformation. A simple, scalable intervention based on teaching evaluation heuristics (rather than correcting specific claims) can improve people's ability to distinguish credible from non-credible news. However, the heterogeneous effects across contexts suggest that interventions may need customization for populations with limited digital familiarity. The decay of effects over time points to a need for reinforcement rather than one-shot interventions.