Scaling up fact-checking using the wisdom of crowds

Contributions

• Demonstrates that small, politically balanced crowds of lay raters can achieve accuracy comparable to professional fact-checkers when rating news article veracity.
• Shows that the average aggregate rating from laypeople correlates significantly (r = 0.62) with professional fact-checker ratings across articles.
• Develops a machine-learning classifier using laypeople aggregate Likert ratings to predict binary true/false article labels with reasonable accuracy (~74–80% AUC).
• Documents that laypeople's accuracy is robust even when crowds are composed of politically imbalanced participants, though performance improves when participants reflect the political spectrum.
• Identifies key individual differences (cognitive reflection, political knowledge) that predict agreement with professional fact-checkers.

Method

The study recruited 1,128 U.S. residents via Amazon Mechanical Turk to rate the accuracy of 207 news articles selected via a stratified random procedure. Participants rated articles on multiple dimensions—truth, misleadingness, falsity, and whether the article was "couldn't be determined"—using Likert scales. Articles were sampled to include diverse topics (e.g., health, politics, crime) and were labeled as truly problematic content flagged by Facebook's internal content moderation systems.

In parallel, three professional fact-checkers from the platform Upwork independently evaluated the same 207 articles, rating each as true, false, or indeterminate. The researchers compared laypeople's aggregate ratings (averaging across randomly sampled subsets of crowd participants) against the professional fact-checkers' categorical verdicts. They also constructed machine-learning classifiers trained on laypeople aggregate Likert ratings to predict whether professional fact-checkers would label an article as "true" or "not true" (binary classification).

Results

The average aggregate laypeople ratings correlated significantly with professional fact-checker ratings: r = 0.62 (range 0.53–0.81 depending on source condition). The Fleiss kappa statistic among the three professional fact-checkers was only 0.51, indicating modest inter-rater agreement even among experts. When predicting whether a professional fact-checker would call an article true, the laypeople-based classifier achieved AUC ≈ 0.83 (95% CI 0.81–0.85), comparable to the classifier's performance on a source condition that revealed the article's source domain. Laypeople predictions remained effective even when the crowd was politically imbalanced (Democratic vs. Republican), though performance degraded slightly. Individual differences in cognitive reflection and political knowledge predicted higher alignment with professional fact-checkers.

Notes

Strengths: The paper tackles a genuine scalability problem in fact-checking and provides empirical evidence that crowdsourcing can be viable. The use of politically balanced crowd composition is thoughtful, and the correlation with professional fact-checkers is reasonably strong. The work is published in a top-tier venue (Science Advances) and the experimental design is clear.

Limitations: The laypeople sample, while geographically diverse, is drawn from Amazon Mechanical Turk and may not represent the general population. The set of 207 articles was sampled in a specific automated way by Facebook and may not be representative of all online misinformation. The study does not assess scalability at the platform level—e.g., the practical cost and latency of recruiting crowds on social media platforms. Inter-rater agreement among professional fact-checkers is modest, raising questions about what "ground truth" means for fact-checking. The paper also does not address the adversarial challenges of scaling—e.g., coordinated attempts to game crowd verdicts or the generalizability to other countries or cultural contexts.

Follow-ups: Future work should test scalability in production settings (e.g., integrating crowdsourced ratings into Facebook's content moderation pipeline), assess cross-cultural generalizability, and investigate defenses against adversarial manipulation of crowd ratings.