Accuracy prompts protect professional content moderators from the illusory truth effect

Contributions

• Demonstrates the illusory truth effect extends to professional content moderators in real-world contexts, not just laboratory populations
• Shows the effect generalizes across non-Western populations (India, Philippines), challenging assumptions that the phenomenon is idiosyncratic to Western cognition
• Identifies and validates a simple, scalable intervention (accuracy prompts) that protects moderators from the effect
• Provides evidence that accuracy mindset interventions work similarly across general and professional populations with varying cognitive styles
• Reveals practical implications for platform safety: exposure effects compound over time and may compromise moderator accuracy, but can be mitigated through design

Method

Three preregistered experiments using COVID-19 news headlines (48 headlines: 24 true, 24 false):

Study 1: Lab-in-field with professional moderators (N = 199 from TaskUs, India and Philippines) Participants rated 16 headlines on interestingness, completed demographics and psychological measures, rated accuracy of the same 16 headlines plus 32 new headlines. Main outcome: accuracy ratings of repeated vs. non-repeated headlines, comparing control (rated interestingness) vs. treatment (rated accuracy during initial exposure).

Study 2: Cross-cultural general population (India N = 997; Philippines N = 1,184) Identical design and procedure to Study 1, conducted online with general population samples to examine cross-cultural generalizability of both the baseline effect and the protective intervention.

Study 3: Lab-in-field replication (N = 239 new TaskUs moderators) Replication of Study 1 design with different sample of professional moderators to confirm treatment effects with fresh subjects.

All studies used Bayesian mixed-effects models (accuracy rating ~ headline veracity × repetition + condition and interactions) with robust controls for covariates (age, gender, education, income, cognitive style, COVID-19 concern).

Results

Baseline illusory truth effect confirmed across all samples:

• Professional moderators: repeated false headlines judged as more accurate (b = 0.10 [0.04, 0.15], 7.1% increase); true headlines unaffected by repetition
• India general population: b = 0.11 [0.06, 0.17], 4.4% increase
• Philippines general population: b = 0.12 [0.08, 0.15], 7.0% increase
• Effect sizes comparable across professional and general population contexts

Accuracy prompt intervention eliminates the effect:

• Professional moderators (Study 1): repetition-by-condition interaction b = −0.15 [−0.28, −0.02]; treatment participants judged repeated false headlines as less accurate than non-repeated ones
• Study 3 replication: b = −0.15 [−0.28, −0.02], confirming treatment robustness
• General population (India): b = −0.14 [−0.22, −0.07]; Philippines: b = −0.16 [−0.22, −0.11]
• Control condition analysis (repeating prior finding): participants judging only true headlines showed baseline repetition effect of b = 0.28 [0.14, 0.42] but no modulation by veracity

Moderation and robustness:

• Effects survived controls for covariates (education, income, analytic thinking, COVID-19 concern, conspiracy mentality)
• Interaction with analytic thinking: moderators with higher dispositional analytic thinking better at discerning true from false (b = 0.33 [0.21, 0.46]) and showed weaker baseline repetition effect; similar pattern in Philippines
• Treatment effect stronger at higher education levels in India (repetition-condition-education interaction: b = −0.10 [−0.18, −0.02])
• Evidence of moderation by income in Study 2 (weaker treatment effect at higher income in India: b = −0.04 [−0.07, 0.00])
• Doubly robust treatment effects estimated via causal forests (professional moderators: b = −0.13 [−0.26, −0.01])

Notes

Strength: Preregistration, multiple replications across populations, lab-in-field design with real moderators, cross-cultural validation showing the effect generalizes beyond Western samples. The intervention is simple (a single prompt) and works at scale in professional contexts. Finds no robust individual-difference moderators of treatment effect, suggesting wide applicability.

Weakness: All samples limited to India and Philippines; cross-cultural scope narrow (two countries). Effect sizes are modest (4–7% in general population, 7% in professionals). Study 3 finds weaker baseline repetition effect in professionals (6%) than Studies 1 and 2, suggesting possible sample variability. Post-treatment measurement of covariates in Study 3 introduces potential bias (mitigated via robustness checks and causal forests, but acknowledged limitation).

Practical implications: Periodic accuracy reminders or "accuracy-focused review cycles" could be incorporated into moderator workflows to counteract exposure effects. The finding that professional moderators remain susceptible despite experience suggests training alone does not prevent the effect. Important for platform scale: if thousands of moderators are exposed to false claims daily, cumulative accuracy degradation could affect moderation quality over time without intervention.