Skip to content

Label quality

Label quality—the reliability and consistency of ground truth annotations—is critical to training robust misinformation detection systems. Models can only learn patterns as reliable as their training signal, making annotation quality a primary determinant of system performance and generalization.

Key challenges

Annotator disagreement: What one expert believes to be false, another might reasonably judge as partially true or context-dependent. High inter-annotator agreement (Cohen's kappa >0.7) is difficult to achieve, especially for nuanced, ambiguous claims.

Expert vs. crowdsource: Expert annotations (from fact-checkers or domain specialists) tend to be more reliable but scale poorly; crowdsourced annotations scale but introduce noise and potential biases from crowd worker demographics and expertise.

Ambiguous and vague claims: Some claims are too vague or depend on interpretation to credibly assess veracity. Including such claims in a dataset introduces irreducible label noise.

Temporal and contextual sensitivity: The veracity of claims can depend on temporal context (e.g., "inflation is rising" is true in some periods but not others). Datasets that obscure this context introduce inconsistent labeling.

Source-based biases: Labeling based on the outlet that published a claim (e.g., marking all articles from a known misinformation outlet as false) conflates content-level falsity with source reputation, introducing confounds.

Labeling approaches in practice

Expert annotation: Professional fact-checkers or domain experts manually review each claim. Highest quality but lowest scale (e.g., PolitiFact-based LIAR dataset: ~300 claims annually).

Crowdsourced annotation: Workers on platforms like Amazon Mechanical Turk label claims. Scales better but requires careful design of guidelines, quality control mechanisms, and aggregation strategies (e.g., majority vote).

Source-based categorization: Automatic labeling based on outlet reputation or algorithmic classification. Scales maximally but introduces systematic biases.

Hybrid: Combining expert and crowdsourced annotations (e.g., professional fact-checkers set ground truth, workers provide secondary labels; or initial crowdsourced labels are reviewed by experts).

Key papers