Annotator Disagreement¶
Disagreement among annotators has traditionally been treated as noise to be minimized through clearer guidelines or aggregation to majority labels. However, in many tasks—particularly those involving subjective judgment, interpretation, cultural context, or genuine semantic ambiguity—disagreement is a feature of the data, reflecting legitimate differences in how people understand the world.
In the fake-news and misinformation detection domain, annotator disagreement arises in tasks like stance detection, implicit-argument identification, and toxicity assessment, where different readers may reasonably draw different conclusions from the same text depending on prior knowledge, cultural background, and interpretation of context.
Modern approaches treat disagreement as signal: instead of collapsing annotations to a majority label, methods model the distribution of votes, learn which items are intrinsically contested, and evaluate models on their ability to capture interpretive variation rather than match a single ground truth.
Key papers¶
- Interpretive Variation and Discourse Structure in Detecting Enthymemes in Controversial Political Discourse — proposes the per-vote consistency metric to evaluate models on how well they capture disagreement; shows that genuine disagreement about implicit premises is real and should be modeled directly.
Related topics¶
- Evaluation metrics for language models (soft metrics, distributional approaches)
- Label quality (what makes annotations trustworthy)
- Annotation Methodology (handling variation in human judgment)