Crowdsourced annotation refers to the use of lay human annotators (rather than professional journalists, experts, or automated systems) to label, assess, or evaluate information—particularly in misinformation detection, fact-checking, content moderation, and source credibility. Systems range from platform-native crowdsourced correction (Twitter's Community Notes, Weibo Annotations) to research benchmarks labeled by crowd workers on platforms like Amazon Mechanical Turk.
Key concepts
Quality variability — Lay annotators differ widely in reliability, expertise, and reliability. Research shows that aggregating many lay annotations can match expert-quality labels, but individual annotator agreement is often poor.
Wisdom of crowds vs. partisan bias — Aggregated crowd judgments on misinformation veracity are more reliable than individual judgments, but partisanship and motivated reasoning shape how different annotators rate the same content. Balanced crowd samples help, but ideological sorting limits what a crowd can achieve.
Platform incorporation — Systems like Twitter's Birdwatch (now Community Notes) and Weibo Annotations embed crowdsourced fact-checking into platforms, creating feedback loops where annotators see each other's contributions and platform engagement (likes, comments) shapes which corrections gain visibility.
Annotation and engagement mismatch — Research finds weak or uncorrelated relationships between annotation quality and audience engagement, meaning corrections that are high-quality receive little visibility while popular annotations may lack evidence.
Key papers
- No Note Is an Island: Probing LLMs' Understanding of Crowdsourced Fact-Checking Environments — benchmarks LLM understanding of crowdsourced fact-checking, finding models systematically underrate annotation quality and fail to predict why high-quality corrections receive low engagement
Related topics
- Fact-checking and corrections — crowdsourced fact-checking as one approach
- Label quality — evaluating the reliability of labels
- Crowdsourced governance — broader systems of platform governance via crowds