Will the crowd game the algorithm? Using layperson judgments to combat misinformation on social media by downranking distrusted sources

Contributions

• Empirical test of crowdsourcing robustness: Validates whether layperson trust ratings can reliably identify misinformation sources and examines whether the "gaming" problem (strategic manipulation when people know ratings matter) undermines crowdsourcing effectiveness.
• Partisan polarization despite source-quality discernment: Shows that across the political spectrum, people rate mainstream sources more trustworthy than fake/hyperpartisan outlets, though partisan differences in trust exist.
• Knowledge treatment effects: Demonstrates that informing participants their responses will influence social media ranking algorithms does not substantially degrade their ability to identify misinformation (knowledge treatment p=0.399), addressing a key concern in practical deployment.
• Familiarity as a predictor of trust: Finds that source familiarity plays a significant role in trust judgments; unfamiliar sources are more likely to be distrusted, though this may reflect appropriate skepticism rather than bias.
• Alignment with professional fact-checkers: Crowdsourced trust ratings align reasonably well with professional fact-checker assessments, validating laypeople as practical judges of source credibility.

Method

Design: Between-subject experimental design, N=1,130 recruited (N=984 completed), sampled via Lucid.

Study conditions:

• Control: Participants told survey responses would inform research understanding
• Treatment: Informed that their responses would inform social media ranking algorithms

Source sample: 89 news domains (later analyzed with full set) across categories:

• Mainstream outlets (CNN, Fox News, NY Times, The Guardian, etc.)
• Hyperpartisan sources (Breitbart, Daily Kos, HuffPost, etc.)
• Fake news sites (worldnewsdaily.com, thealtright.com, etc.)

Measurement: Participants rated each domain's trustworthiness on a scale; outcomes compared between control and treatment conditions, stratified by partisan affiliation.

Results

Source-type effect: Significant main effect of source type (p<0.001): mainstream sources received substantially higher trust ratings than fake or hyperpartisan outlets, regardless of condition.

Partisan asymmetry: Republicans gave lower trust ratings to mainstream sources than Democrats, a difference that persisted in both conditions. However, the partisan gap did not eliminate consensus—all mainstream outlets still rated above all clearly fake sites.

Knowledge treatment: No significant main effect of informing participants their ratings would affect algorithms (p=0.399), suggesting that knowledge of real-world impact does not substantially alter crowdsourcing accuracy.

Polarization measure: Political polarization in trust ratings increased modestly in the treatment condition (those informed ratings matter show slightly more partisan divergence), but this polarization effect was small and did not substantially impair source discrimination.

Familiarity role: Unfamiliar sources were more frequently distrusted, and familiarity predicted trust scores. However, this tendency was adaptive: unfamiliar sources included many actual misinformation outlets, suggesting familiarity heuristics work reasonably well for source assessment.

Notes

This paper addresses a critical practical concern in content moderation: does crowdsourcing work only in the lab, or does it remain effective when scaled and made public? The finding that knowledge treatment does not significantly degrade performance is encouraging for platform design, suggesting that transparent crowdsourcing—where users know their ratings matter—could be deployable. However, the modest increase in partisan polarization under the treatment warrants further investigation into long-term effects and strategic behavior on real platforms.

The paper uses a balanced sampling strategy (equally weighting Democrats and Republicans in aggregation), which may not reflect real-world demographics or platform composition. Future work exploring weighted sampling and cross-platform validation would strengthen the case for implementation.