X-FACT: A New Benchmark Dataset for Multilingual Fact Checking

Contributions

• Large-scale multilingual fact-checking dataset: X-FACT contains 31,189 factual claims (an order of magnitude larger than prior multilingual work) spanning 25 languages across 11 language families, sourced from 85 non-English fact-checking websites via the International Fact-Checking Network.
• Fine-grained veracity labels: Seven-point label scheme (True, Mostly-True, Partly-True, Mostly-False, False, Unverifiable, Other) replacing binary classifications, capturing nuance in fact-checking as practiced by human fact-checkers.
• Robust evaluation framework: Three test sets measure different model capabilities: in-domain (α1, 3,826 claims), out-of-domain (α2, 2,368 claims from different fact-checkers), zero-shot (α3, 3,381 claims in languages unseen during training), enabling diagnosis of overfitting and assessment of cross-lingual generalization.
• Empirical baselines with analysis: Multilingual BERT models using claims only, evidence-based aggregation with web search snippets, and metadata augmentation; findings show models achieve 41.9% F1 on in-domain but degrade to 15.4% on out-of-domain and 16.0% on zero-shot, indicating both source-specific and language-specific overfitting.

Method

Data collection: Claims sourced from 85 fact-checking websites identified via the International Fact-Checking Network (IFCN) and Duke Reporter's Lab, excluding English-language fact-checkers already covered by LIAR and MultiFC. Used Google's Fact Check Explorer (GFCE) and direct website crawling to extract claims, claimant information, URLs, and review dates.

Data processing and harmonization: Addressed two major challenges:

1. Multilingual labels: Manually translated all fact-checker ratings to English.
2. Heterogeneous rating scales: Developed a unified seven-label scheme in consultation with Factly (IFCN signatory), then manually mapped each fact-checker's original labels to this scheme based on their rating definitions. Example: mapped PolitiFact's "Mostly False" to the dataset's "Mostly-False" label.

Filtering: Removed 54 fact-checking websites with highly imbalanced data (>80% single label) and 31 websites with fewer than 60 examples. Final dataset: 31,189 claims.

Train/dev/test split: Selected top 12 languages by claim count (average 1,784 examples per language). Split as: training 75% (19,079 claims, 13 languages), development 10% (2,535 claims), in-domain test α1 15% (3,826 claims), remaining sources form out-of-domain test α2. Zero-shot test α3 (3,381 claims) uses 13 additional languages unseen during training.

Models: Fine-tuned multilingual BERT (mBERT-base, 125M parameters) with three approaches:

1. Claim-Only: Claims as input; simple text classification.
2. Attention-based Evidence Aggregator (Attn-EA): Encodes claims and top-5 Google search snippets via mBERT; dot-product attention computes snippet weights, linear combination aggregated with claim representation.
3. Metadata augmentation (+Meta): Concatenates claim text with key-value pairs (language, website-name, claimant, claim-date, review-date) formatted as "Key: Value".

All models trained multilingual (single model for all languages) due to per-language data scarcity.

Results

Quantitative performance (macro F1 scores, Table 3):

• In-domain (α1): Attn-EA + Meta achieves 41.9% F1 (±1.2), vs. 38.2% for Claim-Only baseline; metadata alone improves Claim-Only to 39.4%.
• Out-of-domain (α2): Best model (Attn-EA + Meta) achieves 15.4% F1, vs. 15.4% for Claim-Only + Meta—evidence provides no improvement and suggests overfitting to in-domain patterns.
• Zero-shot (α3): Claim-Only + Meta (16.7%) outperforms Attn-EA + Meta (16.0%), indicating evidence-based model overfits in-domain data.

Evidence quality analysis: Manual inspection of 100 Hindi examples from α1 found only 45% of search snippets provide sufficient information for classification; hypothetical access to full web pages would enable 83% sufficiency, explaining limited gains from snippet-based evidence.

Cross-lingual transfer degradation: Major performance drops from in-domain (41.9%) to out-of-domain (15.4%, -26.5 points) and zero-shot (16.0%, -25.9 points) reveal both source-specific and language-specific pattern overfitting. Models fail to generalize across fact-checking websites and languages.

English data augmentation experiment (Table 4): Augmenting training with 12,311 PolitiFact claims hurt performance (Attn-EA + Meta: 38.0% on α1, 14.7% on α2), hypothesized because PolitiFact data is politically skewed while X-FACT spans general topics.

Notes

X-FACT addresses a critical gap: most prior fact-checking research focuses on English, despite misinformation being a global phenomenon. The dataset's scale (31K claims across 25 languages) and rigorous label harmonization represent significant contributions. The challenge sets are well-designed: α2 tests domain generalization (different fact-checkers, same languages), while α3 tests linguistic generalization (unseen languages), both revealing model fragility.

The ~40% F1 score on in-domain is notably lower than monolingual English models achieve on LIAR or FEVER, indicating multilingual fact-checking is inherently harder—likely due to language diversity, smaller per-language training sets, and reduced Wikipedia evidence availability for non-English claims. The evidence-based model's failure to improve performance on zero-shot (α3) is particularly important: it suggests evidence retrieved in a target language may be less reliable than evidence for English, or the attention mechanism fails to weight cross-lingual evidence effectively.

Limitations: (1) claim sourcing biased toward fact-checkers with structured websites amenable to crawling; some languages underrepresented; (2) labels aggregated from different fact-checkers with different standards (no inter-annotator agreement reported on label harmonization); (3) zero-shot set uses only 4 languages (Persian, Bengali, Albanian, others from Table 6) due to data availability, limiting language diversity in the zero-shot evaluation. Future work should explore better evidence retrieval in low-resource languages and fine-tuned multilingual models per language.

The dataset has been widely adopted in the fact-checking community and is available at https://github.com/utahnlp/x-fact.