Unbundling Digital Media Literacy Tips: Results from Two Experiments

Contributions

1. Unbundles literacy tips: Previous work (Guess et al. 2020; Epstein et al. 2021) tested 10 tips as a bundle. This paper tests each tip individually to measure distinct effects and avoid conflation with additive treatments.
2. Distinguishes accuracy from sharing behavior: Most tips improved sharing restraint without improving accuracy judgments, revealing different psychological mechanisms.
3. Identifies "near transfer" effects: Only tips targeting specific cues in headlines (source, domain) improved discernment; tips promoting general skepticism did not generalize.
4. Highlights source credibility as a heuristic: Fine-tuned examination of tip × headline interactions shows participants rely on source trustworthiness as a diagnostic feature, not on tips' abstract advice.

The 10 Tips

All tips were originally piloted by Guess et al. (2020) and Facebook's platform intervention. The paper tests:

1. Don't forget to think about accuracy — Accuracy prompt encouraging deliberation about content truth before sharing
2. Seeing is believing — Warning against the illusory truth effect; exposure alone increases belief
3. Be skeptical of headlines — Alert to sensationalism, unusual capitalization, exaggerated claims
4. Consider the source — Check whether the publisher is trusted and has a reputation for accuracy (✓ effective in both studies)
5. Watch for unusual formatting — Notice misspellings, awkward layouts, poor grammar
6. Consider the photos — Be aware that images may be taken out of context
7. Why was this story shared? — Reflect on motives of the author and person who shared it
8. Is the story a joke? — Check whether the story is satire or parody
9. Some stories are intentionally false — General warning about deliberate misinformation
10. Look closely at the website domain — Identify phony URLs mimicking legitimate news sites (✓ effective in both studies)

Method

Two parallel experiments, each randomizing tips independently:

Study 1 (S1):

• Two sub-experiments ran in parallel with accuracy judgments (DV1) and sharing intentions (DV2) as separate dependent variables
• Participants randomly assigned to one of 10 tips or control (20% control, 80% to tips)
• Presented 20 social media posts (10 true, 10 false; balanced partisan lean: pro-Democratic, pro-Republican)
• Measured perceived accuracy (1–6 scale) and sharing likelihood (1–6 scale)
• N = 4,926 (Lucid panel)

Study 2 (S2):

• Modified design: only accuracy measured; 10 treatment groups plus control
• Within each treatment, tips randomly selected and shown together (number of tips varies: 0 tips = control, 1–10 tips shown together)
• N = 2,524 (Lucid panel with pretreatment attention screening)

Headlines: Balanced true/false, pro-Democratic/pro-Republican. False headlines drawn from fact-check websites (Snopes, FactCheck.org); true headlines from mainstream news. Post hoc assessment: 34 crowd raters rated trustworthiness of publishers in headlines (true headlines from trustworthy sources, false from untrustworthy).

Results

Accuracy Discernment (Perceived Accuracy)

Study 1 — Only two tips reached significance:

• "Look closely at the website domain" (β = −0.37, p < .001) — drew attention to source, increased accuracy sensitivity
• "Consider the source" (β = −0.26, p < .001) — similar effect via source-credibility heuristic

No significant effects for:

• Tips emphasizing general skepticism ("Some stories are intentionally false," "Be skeptical of headlines")
• Tips emphasizing critical thinking ("Why was this story shared?," "Is the story a joke?")
• Most other individual tips

Study 2 — Confirms but adds one more:

• "Look closely at the website domain" (significant)
• "Consider the source" (significant)
• Effect of number of tips modest: largest jump from 0 to 1 tip (β = 0.28, p < .01); additional tips show diminishing returns

Mechanism: Regression models pooling Studies 1 & 2 with source-trustworthiness interactions showed that the two effective tips increased sensitivity to source trustworthiness variations rather than increasing general skepticism. Confirms that these tips work via "near transfer"—enhancing attention to a specific diagnostic cue.

Sharing Discernment (Willingness to Share)

Study 1 — 7 of 10 tips significantly increased sharing discrimination:

• "Don't forget to think about accuracy" (β = 0.06, p < .05)
• "Consider the source" (β = −0.13, p < .001) — negative coefficient: reduced sharing of implicitly untrustworthy content
• "Watch for unusual formatting" (β = −0.12, p < .01)
• "Consider the photos" (β = −0.16, p < .001)
• "Some stories are intentionally false" (β = −0.09, p < .05)
• "Look closely at the website domain" (β = −0.17, p < .001)
• "Why was this story shared?" (β = 0.09, p < .05)

Study 2: Did not measure sharing (accuracy only).

Interpretation: The disconnect between accuracy and sharing effects suggests tips work partly as "accuracy primes"—reminding people to be cautious—rather than improving their ability to detect falsehoods. Evidence: tips increase sharing discrimination even when they don't improve accuracy judgments.

Notes

Strengths:

• Pre-registered designs with large samples (N = 4,926, 2,524) rule out most p-hacking concerns.
• Careful bundling/unbundling: cleanly isolates individual tip effects by avoiding combinations.
• Strong internal validity: controlled stimulus set (100 false, 100 true headlines pre-tested).
• Tests both accuracy and sharing outcomes, revealing dissociation between the two.
• Mechanistic evidence (source-credibility interactions) explains why some tips work.

Limitations:

• Ecological validity: Lab task with isolated headlines differs from real social media feeds. Participants rated headlines in sequence, not embedded in a personalized feed with algorithmic ranking, social signals, or recommendations.
• Tip effectiveness context-dependent: Tips work best when the diagnostic feature (source reliability, domain name) is actually diagnostic of falsehood in a given stimulus set. In the study, true headlines came from trustworthy sources and false headlines from untrustworthy ones. In the wild, partisan news outlets may violate this assumption (e.g., Fox News is trustworthy by some, untrustworthy by others).
• Implementation differences from Guess et al. (2020): The earlier study used an "engaged" treatment (participants answered questions about tips after reading them). This study presented tips once before the post. Smaller engagement may explain smaller effects.
• No lateral-reading tips tested: The paper focuses on "vertical" (within-post) cues. It does not test "lateral reading"—opening multiple tabs to investigate sources independently—which professional fact-checkers rely on (Wineburg & McGrew 2019).
• Sharing measured as hypothetical intent, not real behavior. No direct evidence that tips translate to actual platform sharing reductions.
• Study 2 did not measure sharing. The disconnect between accuracy and sharing effects is largely documented in Study 1; Study 2 adds power for accuracy but doesn't replicate the sharing finding.

Why tips for source-credibility work (and why others don't): The paper proposes a "near transfer" vs. "far transfer" distinction:

• Near transfer (source-specific tips): Work by directing attention to a feature of the content that is highly diagnostic of veracity in the stimulus set. Because true and false headlines came from systematically different sources, this tip reliably improves judgment.
• Far transfer (critical-thinking tips): Aim for broad resistance to misinformation across contexts. The paper provides no evidence these tips work in isolation. General skepticism, reflection on motives, and humor detection did not improve accuracy discernment—consistent with research showing that broad interventions to increase general skepticism may backfire by increasing disbelief in true claims too (Hameleers & van der Meer 2023).

Implications for scalability: The finding that only finely targeted tips (source credibility) work limits their deployment. In heterogeneous real-world environments—where source trustworthiness varies by ideology—tips may backfire or fail to transfer. The paper suggests that for tips to work at scale, they must be highly specific to observable cues that are robust diagnostic features across contexts, which is difficult to guarantee.