GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification

Jie Zhou; Xu Han; Cheng Yang; Zhiyuan Liu; Lifeng Wang; Changcheng Li; Maosong Sun

In brief

Most claim verification systems treat evidence independently (via concatenation or pair-wise encoding), missing relationships among multiple pieces of evidence needed to verify complex claims. GEAR proposes a fully-connected evidence graph where an Evidence Reasoning Network (ERNet) propagates information between evidence nodes via attention-based message passing, then aggregates the refined evidence using attention, max, or mean operations. Combined with BERT encoding and evaluated on FEVER, GEAR achieves 67.10% test FEVER score.

Contributions

Graph-based evidence reasoning: Frames evidence as nodes in a fully-connected graph, enabling information propagation between evidence pieces via multi-hop message passing.
Evidence Reasoning Network (ERNet): An attention-based graph layer that computes normalized attention coefficients between evidence nodes and aggregates neighbor features through linear combinations.
Multiple aggregation strategies: Proposes attention (claim-guided), max (element-wise maximum), and mean (element-wise average) aggregators to combine evidence for final prediction.
Integration with BERT: Uses fine-tuned BERT for sentence encoding to capture semantic and logical relationships between evidence and claims.
Strong empirical results: Achieves 67.10% FEVER score on the benchmark test set, outperforming the BERT-Concat baseline by 1.46%.

Method

The system comprises three stages: document retrieval, sentence selection, and claim verification (GEAR).

Document Retrieval & Sentence Selection: Follows the approach of Hanselowski et al. (2018). Entity linking extracts query terms from the claim, retrieves top Wikipedia documents via MediaWiki API, and sentences are ranked by relevance using a fine-tuned ESIM model. A threshold filter removes low-confidence evidence to reduce noise.

GEAR Framework:

Sentence Encoding: Each evidence-claim pair $(e_i, c)$ is encoded with BERT to get evidence representation $e_i$; the claim alone is encoded to get $c$. This paired encoding allows evidence nodes to be guided by the claim during reasoning.
Evidence Reasoning Network: Constructs a fully-connected evidence graph with self-loops (each node attends to itself). The network stacks $T$ layers of attention-based message passing: - Compute normalized attention coefficients $\alpha_{ij}$ between node $i$ and neighbor $j$ using an MLP. - Update node $i$'s hidden state as a weighted sum of its neighbors' features. - After $T$ layers, each evidence node aggregates information from all other evidence through multi-hop propagation.
Evidence Aggregation: Three strategies gather information from refined evidence nodes: - Attention Aggregator: Uses claim representation $c$ to attend over evidence hidden states, weighted by softmax. - Max Aggregator: Element-wise maximum of evidence representations. - Mean Aggregator: Element-wise mean of evidence representations.
Prediction: The aggregated state is passed through a one-layer MLP with ReLU and softmax to produce the final SUPPORTED/REFUTED/NOT ENOUGH INFO label.

Results

Test FEVER score: 67.10% (full pipeline with 1-layer ERNet and attention aggregator)
Outperforms BERT-Concat baseline by 1.46 percentage points
Label accuracy on test set: 71.60%
On a difficult subset requiring multi-evidence reasoning (39% of dev set): 67.44% accuracy with ERNet vs. 65.36% without, demonstrating ERNet's value for complex claims
Evidence-enhanced oracle (assuming perfect upstream retrieval): 77.82% label accuracy, showing the main limitation is upstream evidence retrieval and selection

Notes

Strengths:

Clearly motivated problem: many real claims require integrating information from multiple evidence pieces, and simple concatenation or pairwise methods miss these relationships.
The fully-connected graph and attention-based message passing are intuitive for evidence propagation.
Systematic evaluation on difficult subset isolates the contribution of evidence reasoning separate from upstream retrieval errors.
Case studies (e.g., Al Jardine) illustrate how attention weights identify which evidence nodes are most informative.

Limitations:

Upstream document retrieval and sentence selection account for the majority of errors (~1.4% gap between oracle and actual performance on FEVER). While GEAR improves reasoning over retrieved evidence, the core bottleneck is incomplete evidence retrieval.
ERNet adds only ~1.3–1.7% improvement over simpler BERT-Concat baseline on the full dataset; the gain is larger only on claims that genuinely require multi-evidence reasoning (39% of test cases).
No comparison to other graph-based reasoning methods from the concurrent literature; unclear if fully-connected graphs are optimal or if sparser/structured graphs might work better.

Follow-ups:

Designing better evidence retrieval pipelines (the paper notes Nie et al. 2019's two-hop enhancement as a step in this direction).
Exploring structured graphs (e.g., based on entity/concept overlap) rather than fully-connected graphs.
Extending to long-document reasoning where evidence may span paragraphs rather than single sentences.

How it connects

Uses data from FEVER: A Large-Scale Dataset for Fact Extraction and VERificationThorne et al. · arXiv · 2018 Compares with The Fact Extraction and VERification (FEVER) Shared TaskThorne et al. · ACL 2018 Shared Task · 2018