FEVER (Fact Extraction and VERification) is a large-scale dataset of 185,445 human-verified claims with Wikipedia evidence, designed to benchmark automatic fact verification. The task combines two subtasks:
- Evidence retrieval: Retrieve relevant Wikipedia sentences to evaluate a claim.
- Claim classification: Determine whether the evidence supports (SUPPORTS), refutes (REFUTES), or provides insufficient information (NOT ENOUGH INFO) to verify the claim.
Three-class labels with sentence-level evidence annotation enable both retrieval and classification evaluation. The dataset was introduced by Thorne et al. (2018) and has become a standard benchmark in the field.
Strengths
- Scale: Large, publicly available dataset enabling development of deep neural models.
- Grounding: Claims verified against Wikipedia, a single consistent source.
- Annotation quality: Human verification with inter-annotator agreement reported.
- Reproducibility: Shared task benchmark enabling fair comparison across methods.
Known limitations
- Dataset bias: The dataset contains strong artifacts where claim-only classifiers achieve 61.7% accuracy (far above the 33.3% majority baseline), indicating that models can exploit linguistic patterns (especially negations) rather than reasoning over evidence. 2019-schuster-debiasing-fact-verification identified and proposed methods to address this bias.
- Evidence scope: Verification is limited to Wikipedia; real-world fact-checking uses diverse heterogeneous sources.
- Temporal dynamics: Dataset is static; real claims change validity over time.
- Label granularity: Three-class labels are coarse; many claims have nuanced veracity.
Key papers
- Thorne et al. (2018) — FEVER: A Large-Scale Dataset for Fact Extraction and VERification — Dataset introduction and baseline system.
- Thorne et al. (2018) — The Fact Extraction and VERification (FEVER) Shared Task — Shared task benchmark.
- Schuster et al. (2019) — Towards Debiasing Fact Verification Models — Analysis of dataset bias and proposed debiasing methods.
Related topics
- fact-checking — FEVER is a core benchmark for automated fact-checking.
- evidence-evaluation — FEVER evaluates evidence-based reasoning.
- dataset-bias — FEVER contains identifiable artifacts that allow shortcut learning.
- datasets — Category of benchmark datasets in misinformation research.