Skip to content

Dataset quality

A foundational but often overlooked aspect of misinformation detection research is the quality of the datasets used to train and evaluate systems. High-quality datasets have reliable labels, representative data distribution, and absence of spurious correlations that would allow models to achieve artificially high performance without learning meaningful patterns.

Quality dimensions

Label reliability: How consistent and accurate are the ground truth labels? Differences in annotation guidelines, annotator expertise, inter-annotator agreement rates, and presence of genuinely ambiguous cases all affect label quality.

Representativeness: Does the dataset reflect the real-world distribution of misinformation? Sampling biases (e.g., collecting only from particular outlets or time periods) can make datasets unrepresentative of actual deployment scenarios.

Spurious correlations: Many misinformation datasets contain non-content signals that correlate strongly with the true/false label—specific keywords associated with false claims, temporal patterns in how labels are distributed, or source artifacts—that allow models to achieve high accuracy without learning robust content understanding. Such models often fail to generalize.

Feasibility of evaluation: Can the claims in the dataset actually be credibly assessed for veracity? Some claims are too vague or ambiguous to reasonably fact-check, even with external evidence.

Key papers