David RandContributions
- First meta-analysis directly testing whether fact-checker warning labels work for those distrusting fact-checkers, addressing a critical policy question about intervention generalization.
- Validation of a novel measurement scale for trust in fact-checkers (N=1,000 correlational study).
- Large-scale synthesis across 21 randomized experiments with heterogeneous participant populations and fact-checking contexts.
- Evidence that warning effects persist across political and ideological subgroups, contradicting concerns that polarization fundamentally undermines fact-checking interventions.
Method
The authors conducted a meta-analysis of 21 existing experiments (total N=14,133) in which participants evaluated true and false news posts under random assignment to either: (1) no warning labels, or (2) warning labels on a high proportion of false posts. The meta-analysis focused on two key outcomes: (a) belief in false headlines (measured as agreement/rating scales), and (b) sharing intentions for false headlines. A prior correlational study (N=1,000) validated a measure of trust in fact-checkers to enable participant stratification in the meta-analysis.
Participants with low trust in fact-checkers (operationalized as those in the lowest quartile of the trust-in-fact-checkers scale) were examined as a separate subgroup to test whether intervention effects survive in this population.
Results
- Overall effectiveness: Warning labels reduced belief in false headlines by 27.6% on average (meta-analytic estimate). Sharing of false headlines decreased by 24.7% on average.
- Low-trust subgroup: Even for participants most distrusting of fact-checkers, warning labels significantly reduced belief by 12.9% and sharing by 16.7%. Effects were smaller than the population average but remained statistically significant and practically meaningful.
- Generalization: Effects generalized across diverse experimental contexts, participant demographics, and fact-checking designs, suggesting robustness.
Notes
Strengths: Meta-analysis at scale (21 experiments, ~14k participants) with direct measurement of trust heterogeneity; directly addresses a central policy concern (does polarization/distrust break fact-checking?); effect sizes reported with confidence intervals and heterogeneity estimates. Submitter notes correctly identify this as filling a gap in the Ask GUIDE evaluation of intervention-effectiveness literature: prior meta-analyses (Walter et al. 2020, Chan et al. 2017) focus on generic debunking; this work isolates fact-checker-specific warnings in a polarized context where trust varies widely.
Limitations: Study does not isolate why low-trust individuals still respond to warnings—mechanism remains unclear (are warnings triggering accuracy focus per Pennycook et al. 2021, or something else?). Meta-analysis relies on heterogeneous experimental designs (different label phrasings, different content domains), which may obscure important moderators of trust effects. Real-world deployment may see smaller effects due to habituation, lower engagement, or spillover effects from corrections (Mosleh et al., 2021 showed that being corrected by another user increases subsequent sharing of low-quality content). Time frame of outcomes measured in most included experiments is relatively short (immediate post-exposure); long-term retention is unknown.