Contributions
- Neural semantic matching for evidence retrieval: Develops document retrieval and sentence selection modules that learn deep semantic matching from raw text, significantly outperforming TF-IDF and encoder baselines on FEVER metrics.
- Enhanced claim verification: Augments natural language inference with upstream semantic relatedness scores from evidence retrieval and ontological WordNet features to better integrate evidence selection with claim verification.
- Unified architecture: Formalizes all three FEVER subtasks (document retrieval, sentence selection, verification) as instances of semantic matching, proposing one of the first purely neural fact-checking systems that operates on raw textual evidence from Wikipedia.
Method
The system consists of three homogeneous Neural Semantic Matching Networks (NSMN), each with four layers: encoding (BiLSTM), alignment (cross-sequence attention), matching (BiLSTM over aligned representations), and output (max-pooling + affine projection).
Document Retrieval: Uses keyword matching to filter Wikipedia documents down to a candidate set (~8 documents per claim), then ranks candidates using NSMN. Non-"disambiguative" documents are prioritized; documents requiring semantic understanding (e.g., "Savages (band)") are ranked via NSMN matching between the claim and document title + first sentence. Optionally incorporates Pageview frequency as a reranking feature.
Sentence Selection: From retrieved documents, compares each sentence to the claim using NSMN. Trains with annealed sampling to gradually increase positive-example tolerance during training, balancing recall and precision. Selects top-5 sentences by semantic relatedness score above a threshold.
Claim Verification: Concatenates all retrieved sentences as premise, claim as hypothesis. Enhances base NLI with three token-level feature types: (1) 30-dimensional WordNet ontological features encoding hypernymy/antonymy/distance relationships; (2) 5-dimensional number embeddings; (3) 2-dimensional normalized semantic relatedness scores from upstream document and sentence selection, grounding the verifier in evidence quality.
Results
On the FEVER dev set:
- Document retrieval: NSMN achieves 92.42% oracle FEVER score (k=5, with Pageview reranking); significantly outperforms keyword matching alone (88.86%) and TF-IDF baseline.
- Sentence selection: With annealed sampling, sNSMN achieves 91.19% oracle FEVER; improves evidence recall critical for downstream verification.
- Claim verification: vNSMN achieves 66.14% final FEVER score; ablations show WordNet features (75.7% F1 for "SUPPORTS" vs 74.7% without) and semantic relatedness scores (+1 point F1 on "NOT ENOUGH INFO" vs baseline) both improve performance.
- Final model: Pageview + dNSMN + sNSMN + vNSMN achieves 64.23% FEVER on blind test, competitive with state-of-the-art FEVER shared-task systems and substantially above the baseline (70.2% oracle ceiling with simpler methods).
Notes
Strengths: Elegant unification of three subtasks under a single semantic matching framework. Strong empirical results across all three stages. Ablations clearly demonstrate the contribution of each component (WordNet, relatedness scores, annealed sampling). The paper includes detailed analysis of failure modes and noise tolerance, showing the verifier is robust to imperfect upstream evidence.
Limitations: Relies on keyword matching as first-stage filtering—limiting to exact title/claim overlap may miss relevant documents when terminology diverges. Requires ground-truth evidence during training, limiting applicability to new domains without annotation. The paper acknowledges the model struggles with reasoning requiring world knowledge (e.g., exclusive relations like unique capitals) and cannot capture such constraints from surface text alone. No cross-domain evaluation.
Follow-ups: Later work extends this to handle multi-hop reasoning and incorporate structured knowledge graphs. The FEVER benchmark has spawned numerous improvements in both evidence retrieval and verification components.