Automated Fact-Checking for Assisting Human Fact-Checkers

Contributions

• Comprehensive survey of automated fact-checking technologies from the perspective of professional fact-checkers
• Mapping of the fact-checking pipeline into actionable research problems
• Identification of the disconnect between academic research goals and real-world fact-checker needs
• Analysis of real-world fact-checking systems (ClaimBuster, FEVER, Full Fact, etc.)
• Discussion of practical challenges in deployment and future research directions

Method

The survey is structured around the fact-checking pipeline identified by Vlachos and Riedel (2014):

1. Claim Detection: Identifying statements worth fact-checking from news, social media, or broadcast sources. Systems like ClaimBuster use manually annotated datasets and features (sentiment, named entities, part-of-speech tags) to rank claims by check-worthiness. Recent approaches apply pre-trained transformers (BERT, RoBERTa) and positive-unlabeled learning to handle the label imbalance in observation-based annotation.

2. Previously Fact-Checked Claim Retrieval: Detecting when a new claim matches a previously verified claim, enabling rapid response before the claim spreads. This combines BERT with BM25-style ranking on full fact-checking articles, with follow-up work exploring context (neighboring sentences, coreference resolution, reasoning with Transformer-XL). Multilingual matching is also important because claims often cross linguistic boundaries.

3. Evidence Retrieval: Finding external evidence from trusted sources (Wikipedia, databases, news archives) to support or refute a claim. The task ranges from document-level retrieval (BM25, BERT similarity) to sentence-level extraction and generation of background briefs. The FEVER shared task pioneered evidence extraction paired with natural language inference for verdict prediction.

4. Claim Verification: Determining the truth value of a claim. The paper distinguishes between explainable (reference-based) approaches—verifying against databases, knowledge graphs, or previously fact-checked claims—and non-explainable approaches that aggregate signals from retrieved Web content or social-media propagation patterns. Most systems only achieve high accuracy on narrow claim types (numerical facts with official statistics) or require human review before publication.

Results

The paper's main finding is a partial disconnection between what fact-checkers want and what research delivers:

• What fact-checkers want: Tools to find check-worthy claims, detect previously fact-checked repeats, and retrieve supporting evidence. Most fact-checkers do not believe automated verification will work in the foreseeable future.
• What research prioritizes: End-to-end automated claim verification, which faces credibility and accuracy barriers that prevent adoption.

Real-world systems (Full Fact, BRENDA, Squash, ClaimPortal) increasingly adopt a human-in-the-loop model: monitoring sources, categorizing claims, retrieving evidence, and presenting ranked suggestions to fact-checkers for final judgment.

Notes

Strengths:

• Comprehensive coverage of the entire fact-checking pipeline with real systems as examples
• Clear articulation of the gap between research ambitions and practitioner needs—an important lesson for HCI/AI
• Grounded in interviews with 24 professional fact-checking organizations (via Full Fact's Arnold 2020)
• Practical discussion of why full automation fails (credibility, complexity, trust)

Weaknesses:

• Limited coverage of multimodal verification (deepfakes, manipulated images) despite growing importance
• The survey is tilted toward English-language resources; multilingual challenges are mentioned but not deeply explored
• Lacks quantitative comparison of different approaches (e.g., BERT vs. RoBERTa on check-worthiness)

Open questions:

• How can fact-checking systems be designed to avoid or expose their own biases?
• Can human-in-the-loop systems scale to the global volume of claims without bottlenecking at the human step?
• How do context-dependent claims (e.g., "COVID death rate is rising" — rising where, when, by which metric?) get disambiguated before verification?