Real Solutions for Fake News? Measuring the Effectiveness of General Warnings and Fact-Check Tags in Reducing Belief in False Stories on Social Media

Contributions

1. Empirical test of platform interventions. Evaluates real interventions deployed by Facebook: general warnings about misleading content and two fact-check tag designs ("Disputed," "Rated false").
2. Comparison of warning types. Tests whether specific tags (clearly labeling a claim as false) are more effective than vague general warnings about misleading information.
3. Mechanism of effect. Finds that tags reduce perceived accuracy of false headlines and alter sharing intent independent of whether a general warning is present; results suggest tags work by directly undermining headline credibility.
4. Spillover effects. Identifies an unintended consequence: general warnings also reduce belief in true headlines (12 percentage point decline), suggesting warnings make people skeptical of media more broadly, creating a "tainted truth" effect.
5. Political heterogeneity. Tests whether effects vary for politically congruent vs. incongruent headlines (pro-Trump vs. anti-Trump); finds limited moderation by political preference.

Method

Participants and design: 2,944 Amazon Mechanical Turk workers (54% female, median age 25–34, 55% with bachelor's degree, representative of online survey samples though over-educated and leaning Democratic vs. U.S. population).

Experimental conditions: 2×3 between-subjects design with two factors:

• Tag presence: None, "Disputed" (simulating Snopes/PolitiFact labels), "Rated false" (simulating Facebook's then-current label design)
• General warning: No warning, general warning about misleading articles on social media

Seven conditions total (n = 424–469 per condition).

Stimuli: Participants rated 9 false news headlines: 3 pro-Trump, 3 anti-Trump, 3 true headlines (from mainstream media). Headlines presented as they would appear on Facebook, with experimental manipulations applied.

Measures:

• Perceived accuracy: 4-point Likert scale (1 = "Not at all accurate" to 4 = "Very accurate")
• Sharing intent: likelihood of "liking" or sharing the headline on Facebook (4-point scale)
• Political preferences: Trump approval measured before experimental task

Analysis: OLS regression with robust standard errors clustered by respondent and question; pre-registered on EGAP prior to analysis.

Results

Main effects on false news headlines:

• General warning alone: Decreased perceived accuracy from 1.96 to 1.90 (on 4-point scale), a small but statistically significant effect (−0.08, p < 0.05).
• "Disputed" tag: Reduced the proportion of respondents rating false headlines as "somewhat accurate" or "very accurate" from 29% to 19%, a 10-percentage-point decline (−0.24 on accuracy scale, p < 0.001).
• "Rated false" tag: Reduced accuracy belief from 31% to 16%, a 13-percentage-point decline (−0.34 on accuracy scale, p < 0.001).

Comparative effectiveness: "Rated false" tags were significantly more effective than "Disputed" tags at reducing perceived accuracy of false headlines (p < 0.01), suggesting that explicit, unambiguous language ("false") works better than implicit language ("disputed").

Interaction of tags and warnings: General warning did not meaningfully amplify tag effects; when a general warning was present before seeing headlines, tags still reduced belief in false headlines at similar magnitudes (−0.24 for "Disputed," −0.34 for "Rated false").

Spillover to true headlines:

• General warnings decreased belief in true headlines (−0.12 on accuracy scale, p < 0.01), suggesting a "tainted truth" effect where warnings prime skepticism of all media.
• Tags alone ("Disputed," "Rated false") did not significantly affect belief in true headlines when headlines were untagged.
• Result: Warnings create indiscriminate skepticism; tags are more targeted.

Political heterogeneity: Effects did not significantly vary by whether headlines were politically congruent or incongruent with respondent Trump approval, suggesting warnings and tags work similarly across ideological preferences (contrary to prior research predicting political moderators on belief updates).

Sharing intent: Similar patterns for sharing; tags reduced likelihood of sharing false headlines more than general warnings.

Notes

Strengths:

• Large, pre-registered randomized controlled trial with robust statistical methods (OLS with clustered errors)
• Tests real-world interventions (the "Disputed" and "Rated false" tags were implemented by Snopes and Facebook)
• Systematic comparison of intervention types (general vs. specific, vague vs. explicit)
• Attention to spillover effects (effects on true headlines) often overlooked in fact-checking studies
• Multiple outcome measures (accuracy, sharing intent) strengthen causal inference

Limitations:

• MTurk sample is not nationally representative (over-educated, more politically active, more liberal than U.S. population)
• Headlines presented without article text; real-world exposure often involves reading the full article, which may attenuate tag effects
• No temporal decay measured; effects may dissipate over days or weeks as readers encounter other headlines
• Small sample per condition (n ≈ 400) limits power for detecting interactions by political ideology, though pre-registration helped
• All headlines tested are about Trump; generalization to other political figures or policy domains unclear
• No measurement of whether tags signal source credibility (PolitiFact, Snopes) or platform endorsement of the label

Impact and context: This paper contributed to Facebook's fact-checking labeling strategy during the 2016 election aftermath. The finding that "Rated false" tags are more effective than "Disputed" likely informed platform design choices. The spillover effect (warnings reducing belief in true headlines) is an important qualification on blanket-warning approaches and suggests specificity matters. Published in Political Behavior, a top venue for experimental political science.