Multi-Source Multi-Class Fake News Detection

Authors: Hamid Karimi, Proteek Chandan Roy, Sari Saba-Sadiya, Jiliang Tang
Venue: Proceedings of COLING 2018, Santa Fe, New Mexico
Pages: 1546–1557

TL;DR

Proposes MMFD, a multi-source multi-class fake news detection framework combining automated feature extraction (CNN-LSTM), interpretable multi-source fusion (attention mechanism), and fakeness discrimination. Achieves 38.81% accuracy on LIAR dataset across six classes of fakeness (true, mostly-true, half-true, barely-true, false, pants-on-fire), outperforming prior work that treats fake news as a binary classification problem.

Contributions

• Multi-class framing: Treats fake news detection as a 6-way classification problem reflecting degrees of fakeness, rather than binary true/false, adding nuance to how models discriminate between statements.
• Automated feature extraction: Proposes CNN-LSTM architecture to extract powerful textual features automatically, avoiding hand-crafted feature limitations for short statements.
• Interpretable multi-source fusion: Introduces attention mechanism to combine information from multiple heterogeneous sources (statements, metadata, speaker history, external reports) while tracking source contribution via learned attention scores.
• Fakeness discrimination component: Proposes Multi-class Discriminative Function (MDF), a metric-learning approach that enforces margin between classes while pulling samples toward their class centers.

Method

The MMFD framework has three integrated components:

3.1 Automated feature extraction: Each source is processed independently. For textual sources (statements, metadata, reports), a deep model combining CNN and LSTM extracts features. The CNN captures local patterns (n-grams), while the LSTM captures temporal dependencies across the text. Non-textual sources (speaker history) use a simpler fully-connected network. The key innovation is using deep learning instead of hand-crafted linguistic features (bag-of-words, bigrams, sentiment), which proves crucial for short, adversarial fake news texts.

3.2 Interpretable multi-source fusion: Features from each source are combined via learned attention weights. Linear projections map all source feature vectors to a common dimension \(h\), then an attention mechanism computes scalar weights \(a_i\) for each source \(i\):

\[a_i = \frac{e^{u_i}}{\sum_{l=1}^{k} e^{u_l}}\]

where \(u_i = w^T \tanh(r_i)\) and \(r_i = v_i^T W_i + b_i\). The final fused feature vector sums weighted contributions: \(z_j = \sigma(\sum_{i=1}^{k} a_i r_i)\). Attention weights are learned end-to-end and interpretable — they reveal which sources the model relies on for each statement.

3.3 Fakeness discrimination: Rather than standard cross-entropy loss, MDF uses a learned metric-learning objective combining intra-class and inter-class terms. The intra-class term \(\varepsilon^{\text{intra}}\) minimizes distance of samples to their class center \(c_i\):

\[\varepsilon^{\text{intra}} = \frac{1}{|D|} \sum_{z_j \in D_i} \|z_j - c_i\|_2^2\]

The inter-class term \(\varepsilon^{\text{inter}}\) enforces a margin \(\alpha\) between centers of different classes:

\[\varepsilon^{\text{inter}} = \frac{1}{\binom{m}{2}} \sum_{\substack{i=1 \\ r=1 \\ r \neq o}}^{m} \max(0, \alpha - \|c_o - c_r\|_2^2)\]

Final loss: \(\text{MDF} = \beta_1 \varepsilon^{\text{intra}} + \beta_2 \varepsilon^{\text{inter}}\) combined with cross-entropy. This addresses the challenge that class boundaries become intertwined as the number of classes increases.

Results

Evaluated on the LIAR dataset, which contains 12.8K statements with six fakeness labels and four information sources:

• Statements (S1 only): MMFD achieves 29.06% vs Wang (2017) baseline 27.00%
• Statements + Metadata + History (S1+S2+S3): MMFD 34.77% vs 27.04%
• All sources (S1+S2+S3+S4): MMFD 38.81% vs 27.04% (Wang 2017 no data available on enriched set)
• Ablation studies show: automated feature extraction contributes most (removing it drops 38.81% to 26.06%); multi-source fusion adds 2.91 points; fakeness discrimination adds 3.64 points
• Interpretability: Case study on a speaker with high previous false-statement history shows attention weights correctly identify history source as most informative (0.43), while metadata contributes 0.09

Connections

• Wang 2017 introduces the LIAR dataset and CNN-LSTM baseline; this work extends it to multi-class and multi-source.
• Related to Wang et al. on multimodal embeddings for combining heterogeneous information sources.
• Methodologically related to Ruchansky et al. CSI and Allcott & Gentzkow on fact-checking and stance-detection foundations.
• Attention mechanisms follow Vaswani et al. Transformer and prior work on attention in machine translation.

Notes

Strengths: - Clean framing of fake news detection as multi-class problem with interpretable attention weights; the case study demonstrating attention-score alignment with human intuition (speaker history) is compelling. - Solid experimental design with ablation study isolating the contribution of each component. - Addresses a real problem in binary-classification methods: nuance between degrees of falsehood matters for fact-checking.

Limitations and open questions: - Only evaluated on LIAR (English political statements); generalization to other domains (news articles, social media) unclear. - Attention weights provide interpretability but the model itself remains largely a black box beyond those weights. - Multi-class setup is pragmatic for LIAR's six labels but raises the question of how to handle finer-grained or domain-specific gradations elsewhere. - Dataset split (80/10/10 train/test/dev) is standard but does not address temporal generalization (e.g., new speaker styles or fabrication tactics post-2017).