Crowds Can Effectively Identify Misinformation at Scale

Contributions

• Systematic review of empirical studies (2019–2022) on whether layperson crowds can identify misinformation at scale
• Evidence that crowd ratings of news sources and individual articles correlate highly with professional fact-checker ratings (r=0.86–0.90)
• Analysis of how crowd size, political balance, and measurement approach affect performance
• Discussion of both successful implementations (Twitter's Birdwatch/Community Notes) and open challenges (partisan bias, coordinated attacks, "tyranny of the majority")
• Recommendations for integrating crowdsourced fact-checking into platforms and moderation workflows

Method

The paper is a narrative review synthesizing findings from multiple independent empirical studies conducted by the authors and others. The authors organize their review around two main research questions: whether crowds can identify high- versus low-quality news sources, and whether crowds can rate the accuracy of individual news posts. They then examine conditions under which crowdsourcing works (e.g., politically balanced aggregation, continuous rather than categorical scales) and review real-world deployments.

Results

Source-level judgments. Pennycook & Rand (2019) found that politically balanced crowd ratings of 60 news sources correlated r=0.90 with fact-checker ratings. Epstein et al. (2020) replicated this robustness under conditions where participants were told their ratings would inform platform algorithms (potential gaming incentive), and found the wisdom-of-crowds effect held.

Article-level accuracy ratings. Allen et al. (2021) used headlines flagged by Facebook's internal algorithms and found that crowds of ~20–26 people using 7-point Likert scales achieved accuracy ratings that correlated as strongly with professional fact-checkers as the fact-checkers correlated with each other (r=0.62 benchmark). Godel et al. (2021), using full-article stimulus and categorical ratings, initially reported lower crowd performance; however, re-analysis of their data by the authors showed that when using continuous Likert scales (rather than binary or categorical), crowds still achieved high correlations with fact-checkers (r=0.88) and moderate AUC for predicting fact-checker verdicts.

International and real-world deployment. Arechar et al. (2022) demonstrated the approach generalizes across 16 countries on COVID-19 misinformation headlines. Twitter's Birdwatch (now Community Notes) shows that even when users opt-in (and thus may self-select based on partisan preference), 86% of tweets flagged by Birdwatch users as misleading are also flagged as misleading by at least one professional fact-checker.

Notes

Strengths:

• Rigorous synthesis of converging evidence across multiple independent teams and study designs
• Candid discussion of limitations (source familiarity bias, tyranny-of-majority effects, partisan gaming) rather than overselling the approach
• Clear discussion of measurement choices (continuous vs. categorical scales, aggregation methods) that explain discrepancies in prior work
• Practical recommendations for deployment and integration with existing moderation infrastructure
• Well-grounded in behavioral science (wisdom-of-crowds phenomenon, individual biases that cancel out in aggregation)

Limitations and open questions:

• All studies to date rely on paid crowdworkers (MTurk, similar platforms), not voluntary community participation; scalability to endogenous (opt-in) systems remains uncertain
• Source familiarity and political balance biases remain significant challenges for source-level ratings
• Coordinated malicious attacks and strategic gaming of crowdsourced systems require ongoing monitoring and mitigation
• Crowd ratings may not effectively target specific claims once already widely circulated (article-level ratings work faster, but source-level ratings may be limited in impact)
• The paper does not resolve whether crowd labels are trusted or effective enough to change user behavior (separate question from accuracy)

Follow-ups worth noting:

• How do voluntary (endogenous) crowd systems compare to paid (exogenous) ones?
• What task design, aggregation, and demographic targeting maximize both accuracy and scalability?
• How can platforms mitigate partisan bias in endogenous systems without restricting coverage of legitimately controversial content?