Contributions
- Real-world claim dataset: 4,568 claims from 50 fact-checking organizations (via Google FactCheck and ClaimReview markup), avoiding synthetic or Wikipedia-based claims with known limitations.
- Question-answer decomposition: Claims are annotated with question-answer pairs that explicitly represent the reasoning chain needed to reach a verdict, supported by web evidence with textual justifications.
- Addresses data quality issues: Multi-round annotation process that mitigates context dependence, evidence insufficiency, and temporal leakage — three systematic problems identified in prior datasets.
- Structured evidence representation: Each claim includes evidence sufficiency labels (sufficient/insufficient) and reasoning chains across multiple questions, enabling both question-answering and end-to-end fact verification approaches.
- Baseline and open release: Provides baseline performance using question generation and stance detection, with the full dataset and annotation interface released under CC-BY-NC-4.0.
Method
The annotation process follows three phases with built-in quality control:
Phase 1: Claim Extraction & Normalization. Annotators extract the central claim from fact-checking articles and enrich it with necessary context from the source. This step ensures claims are self-contained and verifiable without reading the full article, addressing the context dependence problem.
Phase 2: Question-Answer Generation & Evidence Sufficiency Check. Annotators generate question-answer pairs that decompose how evidence leads to the verdict. Questions may nest or chain, allowing multi-hop reasoning. After each question-answer pair, a second annotator provides an evidence sufficiency verdict without seeing the original fact-checking article — using only the claim, questions, and answers. Any disagreement triggers re-annotation or discussion.
Phase 3: Quality Control & Verification. A third round of annotation verifies the entire claim with its question-answer pairs, ensuring internal consistency. Annotators provide a final verdict and textual justification. Importantly, the dataset includes four claim types: supported, refuted, conflicting evidence/cherry-picking, and not enough evidence.
Reasoning representation: Claims are mapped to metadata (speaker, publisher, date, location) and labeled by claim type and fact-checking strategy. Question-answer pairs are extractive (copied directly from evidence), abstractive (constructed from evidence), boolean (yes/no with explanation), or unanswerable.
Results
- Dataset scale and composition: 4,568 claims with average 2.60 questions per claim, spanning train/dev/test splits (3,068/500/1,000 claims). Temporal organization prevents information leakage: development and test sets only use evidence published before their claims.
- Label distribution: 27.6% supported, 56.8% refuted, 6.4% conflicting evidence, 9.2% not enough evidence — reflecting the fact-checking articles' bias toward false claims.
- Baseline performance: Question-generation + stance detection achieves moderate F1 scores, showing the task's difficulty. Performance is below human agreement (κ = 0.619), indicating room for improvement.
- Quality metrics: Inter-annotator agreement on evidence sufficiency is κ = 0.619 after multi-round consensus, substantially higher than other fact-checking datasets with this design.
Notes
Strengths: The dataset directly addresses known pain points in fact-checking evaluation: temporal leakage prevention, evidence-verdict alignment, and contextual adequacy. The question-answer decomposition is transparent and interpretable, making it useful for explainable fact-checking and intermediate supervision. Multi-round annotation with explicit quality checks yields high reliability. Real-world claims from established fact-checkers ensure in-scope, actionable data.
Weaknesses: The dataset is relatively modest in scale (4,568 claims) compared to some alternatives. The imbalance toward refuted claims reflects the source articles' selection bias. Single-language (English) and geographically-skewed coverage (primarily US-centric claims in the source material).
Follow-ups: Extensions to multilingual settings, temporal generalization studies (do models trained on earlier years generalize to recent claims?), and end-to-end systems that jointly model question generation and answer retrieval merit further work.