How to Think About Whether Misinformation Interventions Work

Contributions

• Identifies lack of consensus in how misinformation studies evaluate intervention efficacy across different research designs.
• Differentiates between two common approaches: measuring only belief/sharing of false content vs. measuring discernment.
• Demonstrates normative and inferential problems with false-content-only designs: conflate false-content susceptibility with general skepticism; miss interventions that reduce belief in both false and true content equally.
• Proposes discernment (ratings of true − ratings of false) as the primary outcome metric for intervention studies.
• Provides a two-step analytic framework: measure discernment first, then separately examine effects on true and false content to understand mechanisms.

Method

The paper uses normative reasoning and a visual decision framework to argue that intervention efficacy should be evaluated against a clear norm: citizens should maximize accuracy in what they believe and share.

Two research design approaches are compared:

1. False-content-only design: Expose respondents to false claims, measure belief/sharing, rate intervention as effective if it reduces false-content belief regardless of effects on true content.
2. Discernment design: Expose respondents to mixed true and false content, measure discernment as the primary outcome (true − false ratings), rate intervention as effective if it improves the ability to distinguish true from false.

A conceptual scatter plot (Figure 1) illustrates how these approaches diverge: interventions in different quadrants are classified as helpful or harmful depending on the metric used. Re-analysis of seven published studies demonstrates that some interventions appear effective by false-content-only metrics but have null or harmful effects on discernment.

Results

Key findings:

• Measurement divergence: Using false-content-only metrics, interventions in quadrants 2 and 3 (reducing false-content belief) appear effective regardless of their effect on true-content belief. Using discernment, only interventions that improve the ability to distinguish true from false qualify as effective.
• Critical case: Interventions that reduce belief in all content equally (creating general skepticism) are judged as effective under false-content-only designs but have zero effect on discernment, potentially amplifying distrust and harming information-seeking behavior.
• Mechanism matters: Different combinations of effects on true and false content can yield identical discernment results; examining both components separately is essential to understand what the intervention does.
• Preregistration needed: Researchers must explicitly state their normative assumption (e.g., belief in true content is equally valuable as disbelief in false content) and preregister it to avoid experimenter degrees of freedom.

Notes

This is a foundational methodological paper addressing fragmentation in how the field evaluates interventions. The discernment framework resolves a critical ambiguity: prior studies using different metrics reached conflicting conclusions about the same interventions.

The paper's key innovation is normative: making explicit that maximizing accuracy (believing true, disbelieving false) should drive research design, and showing that false-content-only metrics miss this goal. The inferential point is equally important: general skepticism and false-content-specific skepticism are not distinguishable without a true-content baseline.

The two-step analytic approach (discernment first, then components) is now standard in high-impact misinformation intervention work. Pennycook and Rand's subsequent papers routinely preregister this design.

No competing interests declared; supplementary analysis on seven prior studies provided.