Fighting misinformation on social media using crowdsourced judgments of news source quality

Gordon Pennycook; David Rand

doi:10.1073/pnas.1806781116

In brief

Can laypeople reliably judge news source trustworthiness to combat misinformation? Two experiments (n=1,980) show that despite partisan differences, crowds rate mainstream sources substantially more trustworthy than fake news or hyperpartisan outlets, with ratings correlating strongly (r=0.90) with professional fact-checkers. Crowdsourced trust ratings appear viable for ranking algorithms to reduce misinformation spread.

Contributions

Empirical validation of crowdsourcing for source quality: Demonstrates that laypeople's collective judgments of news outlet trustworthiness effectively distinguish mainstream, hyperpartisan, and fake news sources across two large preregistered experiments.
Partisan-neutrality despite political differences: Shows that even when Democrats and Republicans disagree on mainstream media trust, politically balanced aggregation yields high agreement—every mainstream outlet rated more trustworthy than fake/hyperpartisan sources (with one exception).
Strong correlation with expert assessment: Crowdsourced ratings (r=0.90) align with professional fact-checkers' trustworthiness ratings, validating laypeople as reliable judges of outlet reliability.
Role of cognitive reflection: Finds that individuals higher in cognitive reflection (CRT) are better at discerning source quality, particularly among liberals; lack of reflection, not partisan bias, drives poor judgment.
Familiarity as a feature, not a bug: Unfamiliarity with sources predicts distrust; excluding unfamiliar ratings degrades crowdsourcing effectiveness, suggesting familiarity is an appropriate heuristic for source assessment.

Method

Study design: Two preregistered experiments using different sampling approaches:

Study 1: n=1,010 from Amazon Mechanical Turk
Study 2: n=970 from Lucid (nationally representative on demographics)

Source categorization: 60 news websites across three tiers:

20 mainstream media outlets (CNN, NPR, Fox News, NY Times, etc.)
22 hyperpartisan websites (Breitbart, Daily Kos, etc.)
18 fake news outlets (The Last Line of Defense, Now8News, etc.)

Procedure: Participants rated familiarity and trust for each domain using the same language Facebook uses: "How much do you trust each of these domains?" (Not at all / Barely / Somewhat / A lot / Entirely). Study 2 clarified trust as belief that sources "produce truthful news content that is relatively unbiased/balanced."

Validation: Eight professional fact-checkers rated the same 60 sources in Study 2 (interrater reliability ICC = 0.97), enabling direct comparison.

Additional measures: Cognitive Reflection Test (CRT) to measure analytical thinking; political ideology (forced-choice Democrat vs. Republican in Study 1; continuous scale in Study 2).

Results

Category-level findings:

Main effect of source type: Both Democrats and Republicans rated mainstream sources substantially more trustworthy than hyperpartisan (F > 100, p < 0.0001 in all comparisons) or fake news sources (F > 180, p < 0.0001).
Partisan gap on mainstream media: Democrats trusted mainstream outlets significantly more than Republicans (11.5 pp difference in Study 1, 14.7 pp in Study 2). Exception: Fox News, which Republicans trusted more than Democrats (29.8 pp difference in Study 1).
No consistent partisan gap on low-quality sources: Hyperpartisan and fake sites did not show reliable partisan preference patterns across studies.

Individual-level findings:

Politically balanced aggregation: When equally weighting Democrat and Republican ratings, every mainstream outlet except Salon.com (Study 1) received higher trust scores than every hyperpartisan and fake news site. This remained true when restricting to highly ideological participants, across gender and age groups.
Correlation with fact-checkers:
Fact-checkers' mainstream vs. hyperpartisan difference: 55.2 pp (r=0.91, p<0.0001)
Fact-checkers' mainstream vs. fake news difference: 61.3 pp (r=0.93, p<0.0001)
Crowdsourced vs. fact-checker ratings across all 60 sources: r(58)=0.90, p<0.0001 (politically balanced); r=0.92 for Democrats; r=0.73 for Republicans

Cognitive reflection:

CRT positively predicted source discernment (β=0.16, p<0.0001 in both studies).
Conservatism negatively predicted discernment (β=−0.34 to −0.31, p<0.0001).
Interaction effect: CRT's positive effect on discernment was stronger for Democrats (r=0.24, p<0.0001) than Republicans (r=0.08, p>0.1).

Familiarity paradox:

Low overall familiarity: mainstream (60-82%), hyperpartisan (15%), fake news (9-10%).
Excluding unfamiliar ratings reduced crowdsourcing effectiveness: unfamiliar sources were overwhelmingly distrusted (75–87% rated below midpoint), and most unreliable sources were unfamiliar. This suggests people are initially skeptical and trust only after positive experience—making unfamiliarity an appropriate signal of untrustworthiness.

Notes

Strengths:

Rigorous preregistration and large sample sizes (nearly 2,000 participants).
Two independent studies with different sampling strategies (convenience vs. representative).
Expert validation: comparison with professional fact-checkers (ICC=0.97 interrater agreement).
Addresses real concern: does lack of familiarity undermine crowds? Data show unfamiliarity appropriately predicts distrust.
Actionable finding: politically balanced aggregation eliminates partisan bias from output, even when individual participants differ.

Limitations:

All sources rated simultaneously; set composition may have influenced relative judgments.
Ratings subject to "gaming" if domain names are manipulated to sound credible.
Limited to Americans; cross-cultural validation needed.
Study design constrains generalizability: whether results hold in naturalistic social media feeds (where sources aren't pre-selected) is open.
Within-mainstream discrimination is weak (r=0.32 between crowdsourced and expert ratings when considering only mainstream outlets), so crowdsourcing works best for flagging low-quality sources, not ranking reliable ones.

Follow-ups:

Implementation: How do platform algorithms translate trust scores into ranking decisions without over-suppressing novel credible sources?
Gaming resistance: Can trust scores be made robust to coordinated manipulation?
Global applicability: Does crowdsourced trust generalize to non-US media ecosystems?
Temporal dynamics: Do trust ratings adapt as outlets change editorial standards, or do they stagnate?

How it connects

Builds on The Science of Fake NewsLazer et al. · Science · 2018 Related Shifting attention to accuracy can reduce misinformation onlinePennycook et al. · Nature · 2021 Compares with Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoningPennycook & Rand