Where is your Evidence: Improving Fact-checking by Justification Modeling

Contributions

• LIAR-PLUS dataset: An extension of the LIAR dataset with automatically extracted justifications from fact-checking articles, addressing the gap between available training data and human fact-checking evidence.
• Evidence-guided fact-checking: Demonstrates that modeling human-provided justifications—viewed as summary evidence—improves veracity prediction across multiple machine learning paradigms (feature-based and deep learning).
• Comprehensive evaluation: Shows consistent improvements across logistic regression, SVM, and BiLSTM architectures in both binary (true/false) and fine-grained six-way (pants-on-fire, false, mostly-false, half-true, mostly-true, true) classification.
• Justification extraction method: A simple but effective approach for automatically extracting summary justifications from fact-checking articles by identifying "our ruling" or "summing up" sections.

Method

Justification Extraction

The authors extract justifications from PolitiFact articles by:

1. Identifying summary sections marked with headlines like "our ruling" or "summing up"
2. Extracting all sentences in these summary sections
3. Falling back to the last five sentences of the article when no explicit summary exists
4. Filtering out sentences containing the verdict and related keywords

Experimental Setup

Four conditions tested:

• S (Statement): Basic claim representation using word features
• S+M (Statement + Metadata): Enhanced representation with sentiment, emotion, speaker history
• SJ (Statement + Justification): Basic statement plus extracted justification
• S+MJ (Statement + Metadata + Justification): All features combined

Models

Feature-based: Logistic Regression (LR) and Support Vector Machines (SVM) with unigram features and various linguistic lexicons (SentiStrength, EmoLex, LIWC).

Deep Learning: Bidirectional LSTM (BiLSTM) with Glove embeddings, and a parallel architecture (P-BiLSTM) that processes statements and justifications in separate branches before concatenating for classification.

Results

Binary Classification (True/False)

Including justification (SJ, S+MJ conditions) significantly outperformed claim-only baselines:

• Best result: S+MJ with LR achieved 0.69 F1 on validation, 0.67 F1 on test
• Claim-only baseline: S+M with LR achieved 0.61 F1 on validation, 0.61 F1 on test
• Improvement: ~10% absolute F1 gain from adding justification

Six-way Classification

Largest gains on ambiguous classes:

• Best result: S+MJ with P-BiLSTM achieved 0.37 F1 on validation and test
• Claim-only baseline: S+M with BiLSTM achieved 0.27 F1 on validation, 0.25 F1 on test
• Per-class gains: Largest improvements for pants-on-fire and half-true classes (20–30% relative improvement)

Key Findings

The paper demonstrates that:

1. Justification modeling improves performance uniformly across all tested models
2. The parallel BiLSTM architecture capturing statement-justification interactions slightly outperforms single-sequence concatenation
3. Enhanced metadata features (sentiment, emotion, speaker history) provide orthogonal signal to justifications
4. Modeling justification helps most on the hardest classes (ambiguous and extreme labels)

Error Analysis

The authors analyze misclassifications and find:

• Justification helps most when direct lexical signals contradict the claim (e.g., justification explicitly states falsehood)
• Complex justifications without direct textual overlap with claims provide less benefit without additional metadata
• Noise in automatic justification extraction (when "our ruling" sections are missing) limits performance improvements
• Temporal and numeric reasoning remains a challenge for the relatively simple models tested

Notes

This work highlights an important gap in fact-checking research: most systems trained on fact-checking datasets discard the justifications humans use to verify claims. By recovering these justifications through automatic extraction and showing consistent improvements, the paper makes a practical contribution to fact-checking systems.

The justification extraction method is simple but effective, though the authors acknowledge that noise from falling back to last-five-sentence extraction when explicit sections are missing represents a limitation. The results suggest that more sophisticated evidence extraction and ranking methods (as explored in the FEVER shared task) could yield further improvements.

The paper's finding that both metadata and justification signals are valuable aligns with real fact-checking practice: organizational credibility (metadata features) and specific evidence alignment (justification features) both matter for accurate verification. The consistent improvements across architectures suggest the core insight—that justifications are a useful signal—is robust.