Dataset bias refers to systematic, unintended correlations between features and labels in training datasets that allow machine learning models to achieve high accuracy without learning the intended reasoning process. These "shortcuts" or "spurious correlations" are particularly problematic when datasets are constructed through crowdsourcing: annotators often adopt consistent heuristics that leak predictive information, making the dataset vulnerable to models that exploit these patterns rather than performing the actual task.
In NLP, well-known examples include:
- SNLI entailment: Hypothesis-only classifiers achieve ~60% accuracy, far above the majority baseline, because annotators use stereotyped negative examples (e.g., presence of negations strongly predicts contradiction).
- SQuAD reading comprehension: Models achieve high accuracy using just the question, ignoring the passage, due to n-gram overlap heuristics.
- FEVER fact verification: Claim-only classifiers reach 61.7% accuracy because negation phrases like "did not," "refused to," and "failed to" are strongly predictive of the REFUTES label, independent of evidence.
Detecting and correcting dataset artifacts is critical for evaluating whether models truly solve the intended task. Common approaches:
- Analysis: Use Local Mutual Information (LMI) or similar measures to identify n-grams/features strongly correlated with labels.
- Debiased evaluation sets: Construct test sets that explicitly eliminate shortcuts (e.g., symmetric test sets where knowing any feature gives ~50% label probability).
- Data reweighting: Assign higher weights to training examples that contain shortcut features with non-majority labels, flattening spurious correlations.
- Adversarial augmentation: Generate challenging examples that preserve the intended reasoning but eliminate artifact patterns.
Key papers
- 2019-schuster-debiasing-fact-verification — Identifies negation-based bias in FEVER; proposes symmetric test set and re-weighting regularization.
- 2018-poliak-hypothesis-only (cited in paper) — Shows that SNLI is vulnerable to hypothesis-only baselines due to annotation artifacts.
- 2018-gururangan-annotation-artifacts (cited) — Broader study of annotation artifacts across NLP datasets.
- 2019-mccoy-right-for-wrong-reasons (cited) — Syntactic heuristics in natural language inference.
Related topics
- fact-checking — FEVER dataset bias directly impacts fact-checking model evaluation.
- evidence-evaluation — Dataset bias can mask failures in evidence reasoning.
- detection-methods — Dataset artifacts are a fundamental challenge in misinformation detection.