Gamified Inoculation Interventions Do Not Improve Discrimination Between True and Fake News: Reanalyzing Existing Research With Receiver Operating Characteristic Analysis

Contributions

• Critical methodological reanalysis using ROC curves to disentangle discrimination (ability to distinguish true from fake news) from response bias (tendency to call items true/false regardless of veracity), distinguishing signal detection theory's framework from simple pre-post mean comparisons
• Evidence against claimed efficacy of Bad News and Go Viral games across five published studies; shows observed mean-rating changes reflect only conservative responding, not improved discernment
• Demonstration of ROC analysis utility for intervention research; shows how single-point indices (d') carry strong assumptions (equal variance Gaussian distributions) that ROC's non-parametric approach avoids
• Quantification of response bias effects using B''D metric, showing extreme evidence (Bayes factors in hundreds to trillions) that interventions shift response criterion rather than discrimination ability
• Identification of item ambiguity effects — more pronounced effects on discrimination when true/fake news items are similarly ambiguous, masking the lack of true improvement in discriminability

Method

Reanalyzed data from five published experiments across four papers (Roozenbeek & van der Linden 2019, Basol et al. 2020 & 2021, Roozenbeek et al. 2020, Maertens et al. 2021). All studies used pre-post designs measuring perceived reliability of true and fake news items on 1–7 scales. Authors applied:

Signal Detection Theory Framework:

• True news = signal trials; fake news = noise trials
• Participants' subjective evidence mapped to internal continuum
• ROC curves plot hit rates (HR) vs. false alarm rates (FAR) across all scale points
• Discrimination (d') = standardized distance between signal and noise distributions
• Response bias (c) = response criterion placement; B''D = non-parametric bias metric

Analysis:

• Computed area under ROC curve (AUC) for each participant as discrimination measure (ranges 0.5–1.0)
• Calculated B''D for each scale point to assess response bias
• Paired t-tests and Bayes factors (using discrete evidence categories from Jeffreys) comparing pre-test vs. post-test
• Examined whether effects remain robust when correcting for item ambiguity

Results

Roozenbeek & van der Linden (2019, n=13,564):

• AUC increased slightly (M_pretest=.88, M_posttest=.91, d=0.17), but effect nearly negligible despite large sample
• B''D analysis revealed extreme evidence for response bias shift (BF₁₀=1.73×10¹²²)
• Participants became more conservative post-game, rating all items as less reliable
• Improvement appeared only because conservative shift had larger effect on fake news means (which had more room to drop from ceiling)

Basol et al. (2020, n=198):

• Treatment condition: no significant AUC change (M=.73→.75, BF₁₀=0.17, moderate evidence for null)
• Response bias effect: extreme evidence for conservative shift (BF₁₀=48,915.96)
• Researchers' conclusion that Bad News "improved discrimination" rested on omitting true news from analysis, masking response bias

Roozenbeek et al. (2020, Experiments 1–2, n=1,240):

• Set A–A (equivalent items pre/post): no AUC improvement (BF₁₀=0.48); response bias shift (BF₁₀=764.51)
• Set B–B (different items): negligible effects; control condition showed no change
• Item ambiguity effect: effect sizes larger when true/fake items equally ambiguous vs. obviously true

Maertens et al. (2021, n=228):

• Treatment: no significant AUC change across multiple timepoints (BF₁₀=0.01, very strong evidence for null)
• Response bias improved moderately (BF₁₀=5.32×10¹⁵)
• Long-term follow-up showed effect decay; at 9 weeks post-game, no bias change (BF₁₀=0.10)

Basol et al. (2021, Go Viral study):

• Go Viral showed no discrimination improvement in treatment (BF₁₀=2.48) or control (BF₁₀=0.45)
• Similar conservative response bias pattern observed

Cross-Study Pattern: When true news items are obviously true (high ceiling on reliability scale), pre-post mean-rating changes appear large but reflect conservative responding, not discrimination improvement. When items are balanced in ambiguity, effects vanish or reverse.

Notes

Strengths:

• Rigorous statistical methodology: ROC analysis and signal detection theory provide cleaner separation of construct (discrimination) from nuisance factor (response bias) than raw mean ratings
• Comprehensive reanalysis across five diverse studies and design variations (different item sets, timing, control conditions), strengthening generalizability of null findings
• Transparency and reproducibility: all analysis code and data released on Open Science Framework
• Conceptual clarity: explains why prior researchers missed response bias by showing how ambiguous vs. obviously true items shift the apparent effect magnitude
• Methodological contribution: demonstrates non-parametric approach (ROC/AUC) that avoids strong equal-variance Gaussian assumptions required by d'
• Honest treatment of interpretive limits; acknowledges that item ambiguity differences and potential item-specific effects could influence conclusions

Weaknesses:

• Does not experimentally test why games induce conservative responding (psychological mechanism remains unclear; authors hypothesize increased skepticism or wariness)
• Limited to post-hoc reanalysis; cannot rule out all design choices in original studies that might interact with ROC analysis interpretation
• Does not propose alternative intervention designs; conclusion is primarily critical rather than constructive
• Some Bayesian evidence thresholds debated in literature; Jeffreys categories used here are conventional but not universal
• Assumes equal-variance SDT model appropriate; relaxing this assumption might yield different d' estimates (though ROC is more robust)
• Focus on discrimination-versus-bias distinction does not address whether shifting response bias could have downstream psychological value even if discrimination unchanged

Follow-up opportunities:

• Mechanistic investigation: do games increase underlying skepticism (desirable), distrust in media (potentially harmful), or something else? Measure intervening constructs
• Experimental intervention development: if inoculation games fail, test alternative designs (e.g., teaching critical consumption of sources, lateral reading, domain expertise building)
• Longitudinal tracking: do response bias shifts persist, decay, or interact with subsequent media exposure?
• Generalization beyond pre-post: test whether discrimination improves in naturalistic settings post-intervention, not just in experimental tasks with rated stimuli
• Individual differences: explore whether person-level factors (baseline skepticism, cognitive style, political ideology) moderate response bias sensitivity
• Cross-cultural validation: replicate reanalysis in non-Western samples where game effects and response bias patterns may differ