X-FACT is the largest publicly available multilingual benchmark dataset for factual verification, containing 31,189 claims in 25 typologically diverse languages with expert-verified veracity labels. The dataset represents a major contribution to enabling fact-checking systems beyond English.
Dataset Overview
Scale and Languages: 31,189 factual claims sourced from 85 non-English fact-checking websites, covering 25 languages across 11 language families (Indo-Aryan, Romance, Slavic, Germanic, Turkic, Afro-Asiatic, Austronesian, Kartvelian, Albanian, Dravidian, Iranian).
Veracity Labels: Seven-point fine-grained label scheme—True, Mostly-True, Partly-True, Mostly-False, False, Unverifiable, Other—replacing binary classifications to reflect the nuance of professional fact-checking practice.
Label Harmonization: Claims aggregated from heterogeneous fact-checking websites each using different rating scales. Gupta & Srikumar manually developed a unified label scheme in consultation with Factly (IFCN signatory) and mapped each fact-checker's original labels to this scheme.
Data Partitions
| Partition | # Claims | # Languages | Purpose |
|---|---|---|---|
| Train | 19,079 | 13 | Model training |
| Development | 2,535 | 12 | Hyperparameter tuning |
| In-domain (α1) | 3,826 | 12 | Evaluation on same sources/languages |
| Out-of-domain (α2) | 2,368 | 4 | Evaluate generalization to unseen fact-checkers |
| Zero-shot (α3) | 3,381 | 12 | Evaluate cross-lingual transfer to unseen languages |
Language diversity: Training/dev use top 12 languages by claim count. Zero-shot test includes 13 additional languages, ensuring evaluation of cross-lingual transfer. Language families represented include:
- Indo-Aryan: Bengali, Gujarati, Hindi, Marathi, Punjabi
- Romance: French, Italian, Portuguese (3 variants), Romanian, Spanish
- Slavic: Polish, Russian, Serbian
- Germanic: Dutch, German, Norwegian
- Turkic: Azerbaijani, Turkish
- And others: Afro-Asiatic (Arabic), Austronesian (Indonesian), Kartvelian (Georgian), Albanian, Dravidian (Tamil), Iranian (Persian)
Data Collection Process
Claims sourced from non-English fact-checking websites identified via the International Fact-Checking Network (IFCN) and Duke Reporter's Lab. Data extraction via Google's Fact Check Explorer (GFCE) and direct website crawling. Metadata includes claimant, claim date, review date, and source URL.
Filtering: Removed websites with:
- Highly imbalanced data (>80% single label)
- Fewer than 60 examples
- Claims not amenable to automated extraction
- Labels that appeared in claim text itself
Benchmark Results
Multilingual BERT (mBERT-base) baselines:
Best model (Attn-EA + Meta):
- In-domain: 41.9% F1
- Out-of-domain: 15.4% F1
- Zero-shot: 16.0% F1
Performance collapse on out-of-domain and zero-shot reveals strong source-specific and language-specific overfitting. Analysis of 100 Hindi examples found only 45% of search snippets contain sufficient information for classification.
Papers in this wiki using X-FACT
Notes
X-FACT addresses a critical gap in fact-checking research: English-centric benchmarks despite misinformation being a global problem. The dataset's scale and language diversity make it an invaluable resource for multilingual fact-checking. The three-partition evaluation framework (in-domain α1, out-of-domain α2, zero-shot α3) is well-designed for diagnosing overfitting and measuring generalization.
Limitations: (1) sourcing biased toward fact-checkers with structured, crawlable websites; (2) labels aggregated across fact-checkers with different standards; (3) zero-shot set uses only 4 language sources, limiting language diversity in that test split; (4) snippet-based evidence from Google search is limited—access to full documents would likely enable much stronger evidence-based models.
The dataset is freely available at https://github.com/utahnlp/x-fact and has been adopted by the multilingual NLP and fact-checking research communities.