Skip to content
Fact-Checking: A Meta-Analysis of What Works and for Whom

Fact-Checking: A Meta-Analysis of What Works and for Whom

Authors: Nathan Walter, Jonathan Cohen, R. Lance Holbert, Yasmin Morag

Venue: Political Communication, Vol. 37, No. 3, pp. 350–375 (2020) — DOI

TL;DR

Meta-analysis of 30 studies on fact-checking effectiveness finds modest but significant positive effects (d = 0.29) on belief correction. However, effects are substantially moderated by motivated reasoning, political ideology, message design, and context: pro-attitudinal fact-checking works better than counter-attitudinal corrections, visual elements often backfire, and campaign-related statements are harder to correct than routine claims.

Contributions

  • Systematic meta-analysis synthesizing findings from 30 independent empirical studies on fact-checking (14 published, 46.67% unpublished; N = 20,963 across all studies).
  • Quantifies heterogeneity in fact-checking effectiveness and identifies theory-driven moderators that explain when and for whom corrections work.
  • Tests motivated reasoning as a primary mechanism: pro-attitudinal corrections (consistent with preexisting beliefs) are substantially more effective than counter-attitudinal corrections.
  • Evaluates message design factors: lexical complexity significantly reduces effectiveness; visual "truth scales" backfire (d = −0.19) contrary to intuition.
  • Demonstrates partisan asymmetries: Democrats/liberals show stronger corrections from fact-checking (d = 0.43) than Republicans/conservatives (d = 0.17).
  • Identifies context effects: campaign-related statements and election-period messaging attenuate fact-checking effects.

Method

Selection: Five electronic databases searched (Google Scholar, JSTOR, Medline, PubMed, Communication & Mass Media Complete) using key terms including "fact-check," "misinformation," "correction," and "disinformation." Manual reference-list mining and solicitation of unpublished datasets from 17 leading scholars. Inclusion criteria: (1) experimental contrast between fact-checking exposure and control; (2) fact-checking message targeting a political entity; (3) quantitative effect-size estimates. Final sample: 30 studies from 20 research reports (4,816 total excluded; 4,796 excluded for reasons documented in Figure 1).

Coding: Effect sizes computed as Cohen's d (single effect size per study, averaged when multiple estimates reported). Ten theoretically-driven moderators coded at study level: (i) motivated reasoning factors (pro-attitudinal vs. counter-attitudinal, political sophistication); (ii) message design (visual truth scales, lexical complexity measured via Linguistic Complexity Analyzer using four indices); (iii) stimulus characteristics (fact-checker affiliation, partisan nature of claim, campaign timing).

Analysis: Random-effects meta-regression (Comprehensive Meta-Analysis software) using Cohen's d. Heterogeneity assessed via Q-statistic and I² index. Publication bias evaluated via Kendall's Tau and funnel plot asymmetry; trim-and-fill procedure applied to estimate adjusted effects accounting for potential file-drawer bias.

Results

Overall effect: Meta-analytic mean effect size d = 0.29, 95% CI [.23, .36], p = .005. Significant heterogeneity: Q(29) = 207.83, I² = 86.05%, indicating effect sizes vary substantially by study characteristics.

Motivated reasoning effects: - Pro-attitudinal fact-checking (debunking claims opposing preexisting beliefs) significantly more effective than counter-attitudinal corrections: Q(1) = 3.41, p = .04, with pro-attitudinal corrections yielding d = 0.43, 95% CI [.19, .67] vs. counter-attitudinal d = 0.15, 95% CI [.02, .29]. - Political sophistication moderates but does not eliminate counter-attitudinal resistance: higher sophistication associated with weaker belief updating (b = −1.18, SE = .53, Q(1) = 5.04, p = .02). - Partisan asymmetry: Democrats/liberals show larger effects (d = 0.43, 95% CI [.17, .47]) than Republicans/conservatives (d = 0.17, 95% CI [.07, .26]); when examining pro- vs. counter-attitudinal within partisans, Republicans show selective acceptance of pro-attitudinal corrections (d = 0.52) but reject counter-attitudinal ones (d = 0.17; Q(1) = 9.55, p = .001).

Message design effects: - Visual truth scales significantly reduce effectiveness (Q(1) = 6.58, p = .01): visual fact-checks yield d = 0.19, 95% CI [.14, .24] vs. text-only d = 0.31, 95% CI [.23, .39]. - Lexical complexity negatively predicts efficacy (b = −1.43, SE = .43, Q(1) = 12.17, p = .001): each unit increase in composite lexical complexity reduces effect size by 1.43 points. - Message length shows no main effect (Q(1) = 0.30, p = .58); however, interaction with sophistication emerges: longer messages more persuasive for low-sophistication audiences (SE = 27.87, p = .81; curvilinear pattern). - Fact-checking messages refuting entire statements more effective (d = 0.31, 95% CI [.23, .40]) than refuting portions (d = 0.19, 95% CI [.12, .27]; Q(1) = 4.30, p = .03).

Stimulus characteristics: - Campaign-related fact-checking less effective (d = 0.24, 95% CI [.17, .30]) than routine fact-checking (d = 0.38, 95% CI [.26, .51]; Q(1) = 4.07, p = .04). - Fact-checker labels (including source label) vs. no label: inclusion of fact-check label d = 0.18, 95% CI [.11, .26] vs. no label d = 0.32, 95% CI [.24, .40], Q(2) = 6.66, p = .04. - No significant effect of fact-checker affiliation (nonprofit vs. media-housed), logo presence, or partisan source of original claim.

Publication bias: Kendall's Tau = −.30, p = .02, indicating asymmetry; funnel plot inspection suggests smaller studies with larger effect sizes may exist in file drawers. Trim-and-fill adjustment: trimmed average effect d = 0.21, 95% CI [.14, .27], slightly weaker than observed but still significant.

Connections

  • Krause et al. (2020) — Fact-checking as risk communication — complements this meta-analysis with qualitative framework explaining why fact-checking's limited real-world impact may stem from trust deficits, not information deficits.
  • Pennycook & Rand (2017) — Cognitive reflection and fake news susceptibility — demonstrates motivated reasoning and cognitive style as predictors of receptiveness to corrections; mechanisms theorized here.
  • Roozenbeek & van der Linden (2019) — Inoculation game — inoculation-based approach as alternative to post-hoc corrections; complements this finding that reactive fact-checking faces motivated-reasoning headwinds.
  • Pennycook & Rand (2018) — Implied truth effect — documents "truth scale" backfire mechanism (visual ambiguity in fact-check ratings); directly explains H5 finding that visual elements reduce efficacy.
  • Nyhan & Reifler (2015) — When corrections fail — foundational evidence on corrective boomerang effects in politically charged contexts; informs motivated-reasoning hypotheses here.

Notes

Strength. This is the most comprehensive quantitative synthesis of fact-checking research to date. The meta-analysis correctly identifies motivated reasoning as the primary boundary condition on fact-checking effectiveness and demonstrates that simple exposure to corrections is insufficient—political ideology, prior beliefs, and message framing all matter substantially. The finding that pro-attitudinal corrections work much better than counter-attitudinal ones has clear policy implications: fact-checkers should focus on correcting false claims that align with partisan opponents' views (where audience is more receptive) rather than those attacking their own side. The partisan asymmetry is well-documented and not explained away by statistical artifacts. Message-design findings (visual elements backfire, simpler language works better) offer actionable guidance to practitioners.

Limitations. (1) Heterogeneity is very large (I² = 86%), suggesting substantial unexplained variance; moderators tested account for only a fraction of this variation, indicating other unmeasured factors (e.g., media trust, pre-correction belief strength) likely matter. (2) Sample is heavily political; findings may not generalize to non-political fact-checking (e.g., medical, scientific misinformation outside elections). (3) Most studies measure immediate belief change; long-term persistence of corrections is rarely assessed. (4) Publication bias detected via Kendall's Tau; though trim-and-fill adjustment suggests effects survive, true effect may be smaller than observed. (5) Motivations for partisan asymmetry are inferred but not directly tested—are Republicans less responsive because they distrust fact-checkers, or because they hold stronger prior beliefs, or both? (6) Context effects (election season, salience, trust in media) are coded at study level; variation within studies not captured.

Relevance. Essential for researchers designing fact-checking interventions, for platform designers implementing correction systems, and for understanding why fact-checking alone has not measurably reduced misinformation at scale. The paper's emphasis on motivated reasoning and message design reorients fact-checking research away from simple "more facts" approaches toward audience-centric and context-aware strategies.