Generalizing to the Future: Mitigating Entity Bias in Fake News Detection

Contributions

• Identifies entity bias as a critical barrier to fake news detector generalization: the same entity (e.g., Donald Trump) may be strongly correlated with real or fake news in different time periods
• Proposes ENDEF, a model-agnostic debiasing framework based on causal graphs that separately models entity bias (E → Y) and content effects (E → P → Y)
• Demonstrates statistically significant improvements in offline experiments on Weibo (Chinese) and GossipCop (English) datasets using temporal split evaluation
• Reports deployment in a production system, with online improvements on a 10-month 2021 dataset

Method

ENDEF trains two parallel models:

1. Entity-based model ($f_E$): Takes only entity lists as input, explicitly capturing biased entity-to-veracity patterns during training via auxiliary loss $L_E$.
2. Detector ($f_P$): Processes full news content (entities + text), trained jointly with the entity module to learn less-biased features.

During training, the framework aggregates both predictions as $\hat{y} = \sigma(\alpha \hat{r}_P + (1-\alpha) \hat{r}_E)$ with total loss $L = L_O + \beta L_E$. The key insight: at inference time, only $\sigma(\hat{r}_P)$ is used, removing the direct causal effect $E \to Y$. This leaves predictions based on content patterns (writing style, language use) that better transfer across time periods.

The framework is model-agnostic: tested on BiGRU, EANN, BERT, MDFEND, and BERT-Emo as base detectors. The paper also applies token-level data augmentation (drop/mask words and entities) to further boost generalization.

Results

Offline (temporal split: train 2010-2017, test 2018 for Weibo; train 2000-2017, test 2018 for GossipCop):

• Improvements are statistically significant (p < 0.05) across most metrics for Weibo (macro F1 improvements ~2.1%, AUC +1.7%)
• Smaller but consistent gains on GossipCop, attributed to longer news pieces carrying more non-entity signals
• Larger improvements on class-imbalanced metrics (F1fake, spAUC), important for real-world asymmetric costs

Online (10-month 2021 production data, heavily skewed at 40:1 real:fake):

• Relative improvements: BiGRU +2.56%, MDFEND +2.57%, BERT-Emo +0.68% macro F1
• Best model (BERT-Emo w/ ENDEF) deployed to production, processing thousands of suspect pieces daily

Analysis: Figure 3 shows BERT-Emo's average prediction score on fake news improves from 0.49 to 0.58 with debiasing. Case studies confirm the entity-based model captures spurious signals (e.g., "Lionel Messi" biasing toward real, "McDonald's" biasing toward fake).

Notes

Strengths:

• First to explicitly formalize entity bias as a temporal generalization problem, with clear empirical motivation (Table 1)
• Model-agnostic framework—tested on five diverse architectures, all show improvements
• Online deployment evidence is rare and valuable for credibility
• Clean causal framing; inference procedure is simple (drop entity component)

Weaknesses:

• Gains on longer-form English news (GossipCop, avg 606 tokens) are modest compared to Chinese social media (Weibo, avg 120 tokens); suggests method may be less effective on narratively-rich sources
• No comparison to other debiasing approaches (e.g., adversarial methods, data augmentation alone) as separate baselines—ablation limited
• Online results reported as relative improvements only; absolute metrics hidden by business rules, limiting reproducibility assessment

Open questions:

• How does performance scale to even more temporally distant future data (e.g., 2019–2022)?
• Can the framework be extended to multi-modal fake news (images + text)?
• What is the contribution of data augmentation vs. entity debiasing separately?