How Accurate Are Accuracy-Nudge Interventions? A Preregistered Direct Replication of Pennycook et al. (2020)

Contributions

• Transparency in replication science: Preregistered two-stage replication design following SCORE protocols, with independent COS oversight ensuring methodological rigor.
• Effect-size estimation under realistic conditions: Demonstrates ~50% attenuation compared to original study, providing more conservative and evidence-ready effect estimate for policy.
• Mechanism confirmation: Replicates the core mechanism—perceived headline accuracy correlates with treatment impact (r=.57)—supporting the theory of accuracy-redirection.
• Boundary conditions identified: Shows accuracy nudges are less effective for highly attentive participants and may have stronger effects on less-attentive survey respondents; platform recruitment (Lucid vs. Respondi) and headline freshness may moderate effectiveness.
• Debate over true vs. false news impact: Unlike original study, replication shows reduced willingness to share false headlines rather than increased willingness to share true ones, revisiting the preregistered hypothesis that better differentiates mechanisms.

Method

Design and power: Two-stage sequential RCT with preregistration. Stage 1 recruited 701 U.S. participants quota-sampled to be representative on age, gender, ethnicity, and region (90% power to detect 75% of original effect size). Stage 2 added 882 participants for pooled N=1,583 (90% power to detect 50% of original effect).

Procedure: Treatment-group participants rated the accuracy of a single non-COVID headline, then both groups rated sharing likelihood (6-point scale) for 15 true and 15 false COVID-19 headlines. Outcome: truth discernment (difference in sharing likelihood for real vs. false headlines).

Sample differences: Used Respondi (a different panel provider than original Lucid), applied new COVID-related headlines (some original headlines were outdated), and did not exclude participants based on attention checks (though analyzed by attentiveness levels).

Results

Stage 1: No significant interaction between headline veracity and treatment (β=0.0046, p=.67). Replication unsuccessful by SCORE criteria.

Stage 2 (pooled): Significant but attenuated effect. Headline veracity × treatment interaction: β=0.015, p=.017 (uncorrected). Treatment-group discernment d=0.14; control-group d=0.10—a ratio of 1.4× vs. the original 2.8× ratio (50% attenuation).

Mechanism (partially replicated): Perceived headline accuracy correlated significantly with treatment impact (r=.57, p=.001), matching original findings and supporting accuracy-redirection theory.

Hypothesis ambiguity: Unlike the target study (which found increased sharing of true headlines), this replication found participants in treatment were less willing to share false headlines (p=.035, d=0.11) but showed no difference for true headlines (p=.27), aligning with the authors' preregistered hypothesis despite differing from the published results.

Moderators examined:

• Attentiveness: Effect was weaker and non-significant for participants passing all three attention checks (p=.64), suggesting the nudge may be less effective for highly engaged, attentive respondents.
• Partisanship: Significant effect for Democrats (p=.016) and Trump non-voters (p=.003), non-significant for Republicans and Trump voters.
• Medical attitudes: Three-way interaction with Medical Maximizer-Minimizer scores (p<.001), unlike original study.
• No decay over time within task: Marginal evidence for linear decay across headline presentation order (p=.052), suggesting minimal decay within a single survey session.

Notes

Interpretation of partial replication: The 50% attenuation and Stage 1 failure do not negate the core finding but suggest:

1. Effect heterogeneity across contexts: COVID-19 timing, pandemic fatigue, or changed information environment may reduce effectiveness over time.
2. Sample and recruitment effects: More attentive, higher-quality respondents (Respondi's practices) may rely less on attention-redirection nudges.
3. Mechanism is robust, magnitude uncertain: The strong correlation between headline accuracy and treatment impact (r=.57) replicates the theory, even though effect sizes are smaller.

Preregistration discrepancy: The original Pennycook et al. (2020) paper hypothesized that accuracy nudges would reduce false-headline sharing (preregistered), but the published results showed increased true-headline sharing instead. This replication's pattern (reduced false sharing, no effect on true sharing) actually matches the original preregistered hypothesis, raising important questions about how interventions operate and the importance of transparent reporting of hypothesis vs. outcome mismatches.

Implications for policy: SCORE's two-stage design yields a more conservative and policy-appropriate effect size (~50% of the original claim). While smaller than initially hoped, the effect remains statistically significant and directionally beneficial; however, policymakers should expect variation across contexts and test implementations at scale before deployment.

Future directions flagged by the authors:

• Examine accuracy-nudge decay over longer time horizons (days, weeks) using longitudinal designs.
• Test potential moderators (media trust, political polarization) that may explain heterogeneous effects across samples.
• Investigate attentiveness during online surveys as a distinct factor from general attentiveness, which may explain lower efficacy in attentive subgroups.
• Clarify whether intervention impacts false-news or true-news sharing (or both) to better target design.