Do explanations increase the effectiveness of AI-crowd generated fake news warnings?

Contributions

• Empirical evidence that explanations paired with AI-crowd hybrid warning labels increase sharing discrimination relative to labels without explanation
• Demonstration that explanations increase effectiveness particularly for attentive users and those with higher critical reflection scores
• Evidence that trust in the labeling system is not the primary mechanism for explanation effectiveness
• Analysis of moderators including participant attention, age, and political leanings that affect explanation efficacy

Method

The authors recruited 2,512 Americans (1,473 completed the main task) for a between-subjects survey experiment. Participants saw 20 social media "cards" containing headlines — half with false headlines, half with true ones — in random order and rated their sharing intentions on a 6-point scale.

The study employed three conditions:

1. Control: No warning or explanation
2. Label condition: Warning labels (generated by a hybrid human-AI crowdsourcing system) indicating the credibility of the news
3. Explanation + Label condition: Same warning labels plus a brief static explanation of how the labeling system works and the underlying process

After the main task, participants answered detailed factual manipulation checks, questions about the labels they saw, and demographic/individual-difference questionnaires (including critical reflection test, political affiliation, age, gender, education). The analysis used Bayesian mixed-effects models to estimate effects on sharing discrimination.

Results

Main finding: Warning labels increased sharing discrimination (the difference in sharing rates between true and false news) compared to control. Adding explanations increased discrimination further. The posterior probability of the Explanation Label effect on veracity × Label interaction was significant (p < 0.001, preregstered).

Mechanism: Contrary to prior theories, increased trust in the warning labels did not drive the explanations' effect. Instead, participants who received explanations reported better understanding of the labeling process (p=0.169 in the survey; an additional measure of understanding the signal) and perceived the warning labels as more helpful.

Moderation: Explanations were more effective for:

• More attentive participants (higher critical reflection test scores)
• Older participants
• More conservative participants

Explanations showed no significant interaction with education or gender in moderating effectiveness.

Factual understanding: Manipulation checks confirmed that participants in the Label and Explanation conditions understood the warnings differently — some correctly recalled seeing a warning label with an explanation; others recalled only the label alone.

Notes

This work is methodologically sound and fills an important gap: prior research showed that warnings can work but left open the question of whether providing transparency about how a warning was generated affects user behavior. The finding that explanations increase effectiveness without increasing trust suggests design implications for warning systems — transparency about process may matter more than system credibility for user decision-making. The moderation by attention and critical thinking aligns with dual-process theories of belief updating. A limitation is that this is a hypothetical sharing intention task in a survey context, not actual sharing behavior on live social media, though the authors note their framing closely mirrors real platform warning labels.