Contributions
- A novel Kernel Graph Attention Network (KGAT) that integrates neural matching kernels with graph attention networks for fact verification.
- Edge kernels: Conduct fine-grained evidence propagation by identifying key tokens across evidence sentences using kernel-based matching.
- Node kernels: Perform more accurate evidence selection by better matching evidence sentences with claims.
- Extensive analysis showing kernel-based attention produces more focused, interpretable patterns than dot-product attention.
- Strong empirical results: 70.38% FEVER score with RoBERTa, outperforming GEAR and other baselines on both single and multi-evidence reasoning.
Method
The KGAT model constructs a fully-connected evidence graph where nodes represent claim-evidence pairs and edges connect all pairs. It decomposes fact verification into two learned components:
Evidence Propagation with Edge Kernels: For each node, KGAT computes token-level attention using kernels to extract matching features between claim and evidence tokens. A translation matrix of cosine similarities is convolved with K Gaussian kernels (each with learnable mean μ_k and width σ_k) to capture multi-level interactions. Token attention weights are then aggregated via sentence-level attention to update node representations, combining neighbor information.
Evidence Selection with Node Kernels: After edge propagation, node kernels compute a readout representation for each evidence-claim pair. A translation matrix between claim and evidence tokens is processed through kernels to produce selection probabilities. These node-level predictions are combined using the readout module: P(y|G) = Σ P(y|n_p, G) P(n_p|G), where P(n_p|G) is the evidence selection probability.
The model uses BERT to initialize node representations from concatenated claim, Wikipedia title, and evidence. The full architecture is trained end-to-end with cross-entropy loss on the FEVER dataset.
Results
On the FEVER benchmark (185,455 claims):
- KGAT (RoBERTa Large): 70.38% FEVER score, 74.07% label accuracy (test set)
- KGAT (BERT Large): 70.24% FEVER score, 73.61% label accuracy (test set)
- Outperforms GEAR (previous best graph-based approach): +3.28 FEVER points
- Outperforms BERT baselines by ~1–2 FEVER points
Analysis on reasoning scenarios:
- Multi-evidence claims: KGAT-Full achieves 39.23% FEVER (vs. GEAR 37.96%)
- Single-evidence claims: KGAT-Full achieves 78.07% FEVER (vs. GEAR 75.73%)
Attention entropy analysis shows kernel-based attention is significantly more concentrated than dot-product attention, with KGAT node attention having ~2.3× lower entropy than GAT, indicating sharper focus on relevant evidence.
Notes
Strengths:
- The kernel mechanism is well-motivated and empirically sound; entropy analysis convincingly demonstrates that kernels produce more interpretable, concentrated attention patterns than dot-product alternatives.
- Strong performance gains, especially on multi-evidence reasoning (the harder case).
- Clear case study (Table 5) shows KGAT's edge kernels correctly identify "rhythm guitarist" as the missing piece needed to complete reasoning from two evidence sentences.
- Thorough ablations (KGAT-Node, KGAT-Edge, KGAT-Full) isolate the contribution of each kernel type.
Weaknesses:
- Limited to the FEVER dataset; generalization to other fact verification benchmarks (e.g., MultiFC, VitaminC) not explored.
- Computational complexity of kernel computation and graph propagation not discussed; scalability to larger evidence sets unclear.
- The improvement over BERT-based baselines is modest (~1% FEVER), raising questions about whether the added kernel complexity is justified at smaller model scales.
Future directions:
- Extending kernels to the sentence retrieval stage, as mentioned by the authors.
- Evaluating on cross-domain fact-checking tasks to test robustness.
- Analyzing whether kernel-based attention provides benefits in natural language inference beyond fact verification.