On the Efficacy of Accuracy Prompts Across Partisan Lines: An Adversarial Collaboration

Contributions

• Adversarial collaboration design. Two research teams with conflicting prior findings (Pennycook & Rand vs. Rathje et al.) pre-registered competing hypotheses and jointly analyzed pooled data to resolve disagreement.
• Comprehensive multiverse analysis. Examined 70 statistical models varying exclusion criteria, political orientation measures (5 operationalizations), and accuracy prompt types (7 variants) to assess robustness.
• Unified evidence on Republican responsiveness. All 70 models show significant positive effects of accuracy prompts on sharing discernment for Republicans/conservatives, settling earlier debate.
• Conditional partisan moderation. ~50% of specifications show significant moderation by partisan affiliation, with effects stronger for Democrats, but moderation is not consistent across all operationalizations.
• Identity vs. ideology distinction. Partisan moderation is more robust when using binary party identification or voting behavior than continuous ideology scales.

Method

Two research teams (MPR: Martel, Pennycook, Rand; RVV: Rathje, Van Bavel, van der Linden) agreed on a pre-registered analysis plan. The primary data comprised 21 experiments collecting 27,828 sharing decisions on 511 distinct headlines (601,616 total decisions). Experiments varied on:

• Sampling: MTurk (convenience), Lucid (quota-matched), YouGov (nationally representative)
• Topic: Politics, COVID-19
• Accuracy prompts: Evaluation (rate headline accuracy), Importance (emphasize accuracy importance), Norms (peer norms about accuracy), Tips (media literacy), PSA video, Reason (thoughtful sharing)
• Political orientation measures: Party ID, binary Dem/Rep preference, continuous social+economic ideology, Trump 2016 voting, other variants

Analysis used multilevel crossed-effects models predicting sharing intention (true vs. false headlines), with interactions between treatment, headline veracity, and political orientation. Pre-registered hypotheses: (1) MPR predicted significant positive effects for Republicans; (2) RVV predicted effects moderated by partisan affiliation (stronger for Democrats). A third-party moderator (Clark) oversaw analysis to prevent bias.

Results

Main effect (all 70 specifications): Accuracy prompts significantly increased sharing discernment for Republicans/conservatives (min $b=0.017$, $p=0.011$; max $b=0.061$, $p<0.001$), translating to 1.7–6.1 percentage point increases in true relative to false sharing. Effect driven by reduced false-news sharing rather than increased true-news sharing. Supports MPR hypothesis.

Partisan moderation: In 35 of 70 specifications (50%), significant moderation was observed such that effects were stronger for Democrats than Republicans. When using binary party identification or Trump voting measures, moderation appeared in 12–13 of 14 model specifications; when using continuous ideology, moderation appeared in 3 of 14 specifications. Partial support for RVV hypothesis.

• Party ID and Trump voting: robust moderators (12/14 models each)
• Continuous ideology: weak moderator (3/14 models)
• Binary party preference: moderate (8/14 models)
• Conservatism scale alone: weak (3/14 models)

RVV's key pre-registered specification (party ID, evaluation/importance treatments, full sample) showed significant moderation ($p=0.007$ and $p=0.016$). However, when restricting to nationally representative samples (Lucid/YouGov) and evaluation treatments, effects remained significant for both Republicans and Democrats with no significant moderation.

Effect sizes: Republicans $d \approx 0.14$ (nearly identical to Pennycook & Rand's prior estimate), Democrats $d \approx 0.21$ (larger in the current meta-analysis than RVV's 2022 estimate of $d=0.32$, reflecting heterogeneity across headlines).

Notes

Strength: The adversarial collaboration design is exemplary. Two teams with genuinely different prior findings designed a joint study and pre-registered before analyzing pooled data. The 70-model multiverse captures real uncertainty in operationalization and provides transparency about which findings are robust vs. specification-dependent.

On the moderation effect: Both teams agree moderation is not consistently robust. RVV's specific pre-registered predictions were met (party ID + evaluation/importance treatments showed moderation), but the effect is absent or inconsistent in other specifications. The explanation: effects of "evaluation" and "importance" nudges are mediated by perceived headline accuracy; conservatives may rate misinformation as more accurate than liberals (per Garrett & Bond, 2021), reducing the intervention's efficacy for false headlines believed to be true. When moving to representative samples or multiple interventions (which add information beyond accuracy focus), partisan moderation disappears.

Practical significance: Effect sizes are small (1.7–6.1 pp increase in discrimination). Authors argue small effects can scale through networks, but the practical deployment utility of a 3–4 percentage point shift in false-news sharing intentions remains an open question. Future work should examine downstream consequences (actual sharing, engagement, belief change) rather than stated intentions.

Limitation: Samples lean toward MTurk convenience samples early; the representative (Lucid/YouGov) studies come later, and when analysts restrict to those alone, moderation is weaker. Real-world representativeness, political context (study period 2017–2020 spans varying political salience), and generalization to non-experimental settings remain open.