FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information

Contributions

• FEVEROUS dataset: 87,026 claims with evidence from Wikipedia pages (text) and Wikipedia tables, each annotated with supporting, refuting, or insufficient-information evidence.
• Evidence annotation scheme: Explicit annotation of evidence as sentences or table cells with context (page sections, table row/column headers); tracks evidence complexity (single vs. multi-hop).
• Baseline model: Combines entity matching, TF-IDF-based retrieval, table cell extraction, and RoBERTa-based verdict prediction; shows benefits of jointly modeling text and table evidence.
• Analysis of biases: Systematic measurement of dataset artifacts (spurious keyword correlations, word-presence bias), quality control procedures (inter-annotator agreement, gold annotations), and challenge characterization.

Method

The task is formulated as: given a claim, retrieve evidence pieces (sentences or table cells) from Wikipedia and assign a verdict label (Supports, Refutes, Not Enough Information).

Dataset construction:

1. Claims are generated from Wikipedia articles by annotators selecting highlights (4 consecutive sentences or a table) and creating factual or paraphrased claims.
2. A separate annotation stage retrieves evidence: annotators search Wikipedia using ElasticSearch, extract relevant sentences/cells, and label the veracity of each claim given the retrieved evidence.
3. Quality control: 57 annotators for claim generation, 54 for verification; both roles use language-aware screening (50% native speakers, 50% language-aware); inter-annotator agreement measured on random samples (κ = 0.65).

Baseline retrieval pipeline:

1. Retriever: Combines entity matching via TF-IDF and retrieval corpus indexed over Wikipedia articles. For each retrieved article, top k sentences and top q tables are scored using TF-IDF.
2. Cell extraction: For retrieved tables, a RoBERTa model linearizes the table and labels individual cells to extract relevant evidence.
3. Verdict prediction: RoBERTa encoder with concatenated evidence (all retrieved sentences and extracted cells) predicts the final label.

Results

Dataset characteristics:

• Average claim length: 25.3 tokens
• Average evidence: 1.4 sentences, 3.3 cells (0.8 tables)
• Evidence sets balanced: 56% Supported, 39% Refuted, 5% Not Enough Information

Baseline performance:

• Sentence-only baseline: 0.13 score (dev), 0.12 (test)
• Table-only baseline: 0.05 score (dev), 0.05 (test)
• Full baseline (text + tables): 0.19 score (dev), 0.18 (test)

Evidence retrieval:

• Document-level recall@5 coverage: ~69%
• Sentence/table recall: 0.55% for sentences, 0.74% for tables (when gold table is provided)
• Cell extraction from retrieved tables: 0.69 recall, 0.74 F1

Verdict prediction (oracle evidence):

• RoBERTa model achieves 0.89 (Supported), 0.87 (Refuted) F1
• NLI pre-training improves results; NLI + NEI sampling reduces artifact dependency

Notes

Strengths: First large-scale dataset combining structured (tables) and unstructured (text) evidence for fact verification; rigorous quality control with inter-annotator agreement tracking and bias measurement; detailed analysis of annotation process and challenge taxonomy; baseline establishes strong empirical foundation.

Limitations: Retrieval oracle (gold page selection) is unrealistic; end-to-end retrieval performance is substantially lower (0.18 vs. 0.29 with oracle). Dataset focuses on Wikipedia; real-world claims often involve sources outside encyclopedias. Annotation complexity results in lower inter-annotator agreement (κ = 0.65) compared to simpler fact-checking tasks. Limited exploration of cross-dataset generalization.

Impact & follow-up: FEVEROUS has established itself as a standard benchmark for joint reasoning over text and tables. Follow-up work should explore more sophisticated retrieval-verification joint training, cross-lingual extensions, and application to real-world fact-checking workflows.