Skip to content

Crowdsourced annotation

Using lay human annotators to label data, assess content quality, or generate fact-checks

Crowdsourced annotation refers to the use of lay human annotators (rather than professional journalists, experts, or automated systems) to label, assess, or evaluate information—particularly in misinformation detection, fact-checking, content moderation, and source credibility. Systems range from platform-native crowdsourced correction (Twitter's Community Notes, Weibo Annotations) to research benchmarks labeled by crowd workers on platforms like Amazon Mechanical Turk.

Key concepts

Quality variability — Lay annotators differ widely in reliability, expertise, and reliability. Research shows that aggregating many lay annotations can match expert-quality labels, but individual annotator agreement is often poor.

Wisdom of crowds vs. partisan bias — Aggregated crowd judgments on misinformation veracity are more reliable than individual judgments, but partisanship and motivated reasoning shape how different annotators rate the same content. Balanced crowd samples help, but ideological sorting limits what a crowd can achieve.

Platform incorporation — Systems like Twitter's Birdwatch (now Community Notes) and Weibo Annotations embed crowdsourced fact-checking into platforms, creating feedback loops where annotators see each other's contributions and platform engagement (likes, comments) shapes which corrections gain visibility.

Annotation and engagement mismatch — Research finds weak or uncorrelated relationships between annotation quality and audience engagement, meaning corrections that are high-quality receive little visibility while popular annotations may lack evidence.

Key papers

Papers on this topic