Fact Extraction and VERification Over Unstructured and Structured information

Paper: Aly et al. (2021)

Size: 87,026 human-annotated claims

Evidence types:

Unstructured: Wikipedia article sentences
Structured: Wikipedia tables (cells)

Description

FEVEROUS is a large-scale benchmark dataset for fact verification that combines both unstructured text evidence (Wikipedia sentences) and structured evidence (Wikipedia table cells). Each claim is annotated with supporting, refuting, or insufficient-information evidence, along with explicit context including page sections, table row/column headers, and cross-references between evidence items.

The dataset addresses a gap in existing fact-checking benchmarks, which typically focus on either text or tables but not both. FEVEROUS enables research on joint reasoning across multiple information modalities, which is essential for real-world verification workflows that must leverage encyclopedic knowledge in both forms.

Dataset Statistics

Total claims: 87,026
Supported: 49,115 (56%)
Refuted: 33,669 (39%)
Not Enough Information: 4,242 (5%)
Evidence per claim: 1.4 sentences on average, 3.3 cells (0.8 tables)
Evidence sets: 44,963 sentence sets, 28,760 table sets, 24,667 combined
Train/Dev/Test split:
Train: 71,291 claims
Dev: 7,890 claims
Test: 7,845 claims

Annotation Process

Claim generation: Wikipedia articles are highlighted (4 consecutive sentences or a full table); annotators generate claims that are factual, paraphrased, or mutated versions of the highlighted content.
Evidence retrieval and verification: A separate set of annotators retrieves evidence from Wikipedia using guided search, selects relevant sentences/cells, and assigns veracity labels (Supports/Refutes/Not Enough Info).
Quality control: 57 annotators for claim generation, 54 for verification; language-aware screening (50% native English speakers); inter-annotator agreement κ = 0.65; claims with disagreement resolved via gold standard annotation.

Known Properties

Complexity: Claims vary in complexity: Type I uses single highlights, Type II combines information from multiple highlights, Type III involves mutations or rephrasing. Evidence requirements range from single sentences to multi-hop evidence across tables and text.
Balanced coverage: Dataset maintains roughly equal proportion of supported and refuted claims to reduce label artifacts.
Quality checks: Analysis of potential dataset artifacts using normalized PMI measurements; identified minor correlation between word presence and label, but controlled through sampling and model design.

Baselines

Sentence-only baseline: TF-IDF retrieval + RoBERTa classification = 0.13 score (dev), 0.12 (test)
Table-only baseline: TF-IDF + table cell extraction + RoBERTa = 0.05 score (dev), 0.05 (test)
Full baseline (text + tables): Combined retrieval + RoBERTa = 0.19 score (dev), 0.18 (test)

Oracle evaluation (with correct evidence) shows RoBERTa can achieve 0.89 F1 on Supported and 0.87 F1 on Refuted verdicts, indicating that the main challenge is evidence retrieval rather than classification given perfect evidence.

Use Cases

Training and evaluating end-to-end fact verification systems that must handle multiple evidence types
Benchmarking evidence retrieval across structured and unstructured sources
Studying multi-hop reasoning over tables and text
Developing table understanding and cell-selection models
Analyzing annotation quality and dataset artifacts in fact-checking

feverous — FEVER: 185K text-only claims verified against Wikipedia (foundational benchmark)
tabfact — 118K statements verified against Wikipedia tables
HoVER: 26K claims requiring many-hop reasoning

Citation

@inproceedings{aly2021feverous,
  title={FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information},
  author={Aly, Rami and Guo, Zhijiang and Schlichtkrull, Michael and Thorne, James and Vlachos, Andreas and Christodoulopoulos, Christos and Cocarascu, Oana and Mittal, Arpit},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2021}
}

FEVEROUS