Contributions
- Introduces HoVER, a large-scale many-hop fact extraction and verification dataset with 26k claims
- Develops a three-stage collection pipeline: claim creation from HotpotQA, claim mutation to vary reasoning difficulty, and labeling with SUPPORTED/NOT-SUPPORTED/NOT-ENOUGH-INFO judgments
- Shows that most claims (3/4) require evidence from multiple Wikipedia articles, with up to four documents involved
- Evaluates baseline systems revealing significant performance degradation as the number of reasoning hops increases
Method
The collection process follows three stages. Stage 1: Claim Creation starts with question-answer pairs from HotpotQA and transforms them into claims validated by crowd workers. At least three annotators must agree on validity, ensuring quality 2-hop claims.
Stage 2: Claim Mutation extends 2-hop claims to 3-hop and 4-hop claims using automatic and crowd-based methods. Four mutation types are applied: making claims more general/specific, automatic word substitution using BERT, and automatic entity substitution using named entity recognition. Crowd workers validate and extend the chain of evidence.
Stage 3: Claim Labeling assigns each claim a label—SUPPORTED, REFUTED, or NOT-ENOUGH-INFO—using only the supporting documents without external knowledge. Claims where at least three annotators agree are retained.
The baseline system follows a retrieval-and-verification pipeline: TF-IDF or neural-based document retrieval, sentence selection from top documents, and BERT-based claim verification.
Results
The dataset contains 26k claims split into train (15,023), dev (4,000), and test (4,000) examples. Document retrieval performance drops significantly with increased reasoning hops: TF-IDF achieves 16.53 EM@5 for 2-hop claims but only 4.55 for 4-hop claims. The best baseline (BERT + Oracle) reaches 81.2% claim verification accuracy when complete evidence is provided, but only 67.6% when using retrieved evidence. Human performance is 88% on the full pipeline. These results show that even high-quality document retrieval and semantic matching methods fail to capture the complex reasoning required as hopping increases.
Notes
HoVER's strength is its explicit modeling of reasoning complexity through the many-hop structure and clear experimental demonstration of model brittleness. The annotation process is well-designed with validation gates at each stage. A weakness is the reliance on Wikipedia as the evidence corpus, which limits generalizability to other domains. The paper would benefit from deeper analysis of which types of multi-hop reasoning patterns models struggle with most.