Evidence evaluation and reasoning refers to the core task in fact-checking: determining whether a claim is true, false, or indeterminate given specific evidence, rather than relying on independent knowledge or heuristics. This is the intended behavior of automated fact verification systems, though dataset biases often allow models to shortcut this reasoning.
The challenge encompasses:
- Evidence retrieval: Finding the relevant evidence to evaluate a claim.
- Evidence reasoning: Determining whether retrieved evidence supports, refutes, or is neutral to the claim.
- Multi-hop reasoning: Combining evidence from multiple sources when no single document is sufficient.
- Temporal reasoning: Handling claims whose truth value changes over time (e.g., "Halep failed to ever win a Wimbledon title" was true until 2019).
- Open-world verification: Reasoning over evidence from diverse, heterogeneous sources (Wikipedia, news, social media, knowledge graphs) where reliability and coverage vary.
The quality of evidence evaluation is a critical evaluation metric for fact-checking systems. Many datasets inadvertently allow models to ignore evidence due to artifacts, making debiased evaluation essential.
Key papers
- 2019-schuster-debiasing-fact-verification — Shows that FEVER-trained models fail on symmetric test sets; proposes regularization to improve evidence-dependent reasoning.
- Thorne et al. (2018) — FEVER: A Large-Scale Dataset for Fact Extraction and VERification — Foundational dataset combining evidence retrieval with claim classification; shows baseline models struggle with evidence-aware reasoning.
- Jiang et al. (2020) — HoVER: A Dataset for Many-Hop Fact Extraction And Claim Verification — Introduces many-hop reasoning challenge; shows semantic matching models degrade significantly with reasoning complexity.
Related topics
- fact-checking — Evidence evaluation is the core task in automated fact-checking.
- dataset-bias — Dataset artifacts often mask failures in evidence reasoning.