Melisa Basol
Jon Roozenbeek
Sander van der LindenStudy design
The authors employed a 2 (Bad News vs. Tetris control) × 2 (pre-post) mixed design with N=196 (197 recruited, 196 analyzed). Participants from Prolific Academic (58% male, modal age 18–24, 61% liberal, 80% white) completed a survey rating the reliability of 18 fictitious Twitter posts on a 7-point scale before and after gameplay, along with confidence ratings on each judgment. Participants were randomly assigned to play either the Bad News game (~15 minutes) or Tetris (control for gamification effects).
The 18 test items covered six misinformation techniques (impersonation, emotional language, group polarization, conspiracy theories, discrediting opponents, trolling) — three items per technique. All items were explicitly fictional to avoid memory confounds and allow controlled isolation of technique recognition. A plug-in embedded the game in Qualtrics so pre-post testing occurred outside the game environment, enhancing ecological validity.
Key findings
Reliability judgments: The inoculation group showed significantly larger decreases in perceived reliability of fake news items compared to controls (M = −0.45 vs. M = −0.09, d = 0.60, 95% CI [−0.52, −0.19], F(1,196) = 17.54, p < .001). This effect matches the d = 0.52 reported in the original Roozenbeek & van der Linden (2019) study despite using a randomized controlled design, multiple items, and external testing — methodological improvements over the original.
Confidence in judgments: Post-pre difference scores in confidence were significantly higher in the inoculation group (M = 0.22 vs. M = −0.06, d = 0.52, F(1,196) = 13.49, p < .001). This advances prior theory by directly measuring attitude certainty — a dimension of attitude strength theorized to be critical for resisting persuasion.
Conditionality on correct updating: The confidence boost occurred only among the 71% of inoculation-group participants who broadly updated their reliability judgments in the correct direction (M = 0.29 vs. control M = −0.02, d = 0.31, t(126) = 3.37, p < .01). Those who did not change or changed in the wrong direction showed no confidence gain. This dissociation is theoretically important: confidence bolstering occurs when learning is genuine, not when participants apply the inoculation illogically.
No ideological moderation: Political ideology (measured 1=very conservative to 7=very liberal) showed no main effect or interaction with treatment condition, suggesting broad-spectrum effectiveness across the political spectrum, consistent with the original study. However, the smallest effect was on the polarization badge, possibly due to ideologically congruent decision-branching within the game.
Badge-level heterogeneity: Effect sizes for reliability judgment improvement ranged from d = 0.14 (polarization) to d = 0.58 (discrediting), mirroring the range in Roozenbeek & van der Linden (2019). Confidence effects ranged from d = 0.23 (discrediting) to d = 0.49 (emotional language).
Notes
Strengths: The addition of attitude certainty as an outcome measure directly addresses a theoretical gap in the original Roozenbeek & van der Linden study. The finding that confidence increases only among correct updaters is important — it shows the intervention produces learning, not mere response bias (rating everything as less reliable). The randomized controlled design with Tetris as an active control addresses a key methodological limitation of the original. Preregistration and data availability enhance credibility.
Limitations acknowledged by authors: (1) Tetris controls for gamification modality but not cognitive demand; future work should compare against passive fact-checking or traditional critical-thinking interventions to isolate the advantage of active gamification. (2) The study measured simulated Twitter posts pre-post, not real news consumption behavior or social-media sharing, so generalization to real-world contexts remains uncertain. (3) About 29% of inoculation participants engaged in "contrary updating" (shifting judgments in the wrong direction); mechanisms underlying this non-response warrant future investigation. (4) No longitudinal follow-up; inoculation effects are known to decay over time without reinforcement. (5) Prolific sample is not representative of the UK population, and US/non-Western generalization is not tested.
Theoretical contributions: The certainty-confidence dissociation refines inoculation theory by showing that attitude-strength mechanisms (certainty) co-occur with improved judgment accuracy only when participants genuinely learn, not when they apply inflexible heuristics. This supports Tormala & Petty (2004) on certainty as a mediator of resistance, and opens questions about whether confidence reflects genuine discrimination ability or willingness-to-defend beliefs.
Practical implications: The finding that a 15-minute game produces medium-to-large effect sizes, works across the political spectrum, and increases belief in one's own judgment capacity suggests high scalability. Deployment at scale (YouTube, social media) with reinforcement could overcome the noted time-decay problem.