Skip to content

Generating Fact Checking Explanations

Authors: Pepa Atanasova, Jakob Grue Simonsens, Christina Lioma, Isabelle Augenstein

Venue: arXiv, 2020 — arXiv:2004.05773

TL;DR

This paper presents the first study on automatically generating explanations for fact-checking veracity predictions. The authors show that jointly optimizing both veracity classification and explanation generation improves system performance. A multi-task model trained on the LIAR-PLUS dataset produces better coverage and explanation quality than extractive baselines, and demonstrates that veracity explanations can be successfully automated without explicit human justifications during inference.

Contributions

  1. First systematic study of generating veracity explanations from fact-checking data, moving beyond classifying verdict alone.
  2. Demonstration that veracity classification performance can leverage information from generated explanations when trained jointly.
  3. Finding that joint training on veracity prediction and explanation generation objectives outperforms separate task optimization.
  4. Manual evaluation framework assessing explanation quality across coverage, non-redundancy, non-contradiction, and overall coherence.

Method

The paper proposes two explanation approaches:

Explanation Extraction: Greedy sentence selection from fact-checker ruling comments to identify oracle sentences matching human justifications. Uses a DistilBERT-based architecture with task-specific feed-forward layers to predict which sentences should be selected (k=4 on average).

Joint Explanation-Veracity Model: Multi-task learning architecture using DistilBERT with cross-stitch layers to produce separate representations for explanation generation and veracity prediction. The model: - Extracts top-k sentences from ruling comments using the explanation model (confidence-scored) - Jointly predicts both a veracity label from six classes (true, false, half-true, barely-true, mostly-true, pants-on-fire) and generates explanatory text - Optimizes a weighted combination of cross-entropy losses for both tasks during training

Results

On the LIAR-PLUS dataset (10,146 training, 1,278 validation, 1,255 test claims from PolitiFact):

Automatic Evaluation (ROUGE-F1 scores): - Explanation extraction baseline (Explain-Extractive): 35.70 validation, 31.58 test - Joint model (Explain-MT): 35.13 validation, 30.93 test - Both substantially outperform a feed-forward baseline (≤24)

Veracity Prediction: - Joint model achieves 0.443 macro F1 on 6-class task, outperforming separate veracity-only models - Shows that jointly trained systems better capture salient information for both tasks

Manual Evaluation (80 examples, 3 annotators): - Extractive explanations rank highest on coverage and overall quality - Jointly trained explanations achieve better non-redundancy and improve veracity prediction - Low inter-annotator agreement (κ=0.26 for coverage, 0.18 for non-redundancy) indicates task difficulty

Connections

Notes

Strengths: First rigorous study of veracity explanation generation; joint training approach is well-motivated; manual evaluation captures non-obvious quality dimensions beyond ROUGE metrics; dataset construction from real fact-checking sites grounds work in practice.

Limitations: Joint model underperforms extractive baseline on automatic metrics despite human-rated improvements, suggesting ROUGE may not capture explanation quality; low inter-annotator agreement indicates annotation ambiguity; LIAR-PLUS claims are relatively short (39 sentences average in ruling comments). Experiments limited to single dataset and PolitiFact domain.

Follow-up opportunities: Testing on broader fact-checking platforms and languages; investigating why joint training improves interpretability without fully matching extractive ROUGE scores; generating free-form explanations rather than sentence selection; incorporating veracity predictions during explanation inference.