Automated Fact Checking: Task formulations, methods and future directions

James Thorne; Andreas Vlachos

In brief

Survey of automated fact checking from NLP and related fields, unifying disparate task formulations and methodologies. Proposes evidence as a distinguishing factor among approaches and identifies open challenges in building systems that can verify claims at scale using textual, structured, and social evidence sources.

Contributions

Surveys automated fact checking research across NLP, machine learning, knowledge representation, and journalism communities
Unifies terminology and task definitions across disciplines that had developed independently with inconsistent concepts
Identifies and formalizes three dimensions of fact checking systems: task inputs (claims, evidence format), evidence sources (text, knowledge graphs, Wikipedia, social media), and task outputs (labels, rankings, justifications)
Reviews major datasets and benchmarks (PolitiFact, Fake News Challenge, FEVER, HeroX, etc.)
Categorizes existing methods: supervised text classification, relation extraction, RTE-based models, knowledge base approaches, rumor detection
Highlights evidence justification as critical but often neglected in automated systems, contrasting with how human journalists work
Proposes open research challenges: multi-modal evidence, handling open-world knowledge, document-level verification, evidence generation and justification

Method

The survey reviews automated fact checking by systematically covering:

Task formulations: Distinguishes between verification (assessing if evidence supports a claim) and fact checking (judging veracity). Formalizes fact checking task via three axes:

Inputs: Subject-predicate-object triples (requiring entity disambiguation), textual claims (short sentences from web sources or documents), or full documents
Evidence sources: Knowledge graphs (queried with rule-based reasoning), encyclopedic text (Wikipedia), verified news articles, and social network information (rumor veracity prediction via network topology)
Outputs: Binary labels (true/false), multi-point scales (true, mostly-true, half-true, false), rankings of evidence with confidence scores, or structured justifications

Related methods: Supervised text classifiers on linguistic features, Recognizing Textual Entailment (RTE) models, relation extraction for entity linking, knowledge base scoring via graph topology or rule-based inference, speaker profiling (using originator's history of claim veracity), and claim matching against verified repositories.

Limitations and open questions: Highlights that most approaches either ignore evidence or use surface-level heuristics; knowledge graphs cannot capture all feasible facts; systems rarely justify their verdicts; and multi-modal evidence (images, video) is largely unexplored despite journalist reliance on it.

Results

No single empirical results section (this is a survey), but the paper reviews published benchmarks:

PolitiFact dataset (Vlachos & Riedel 2014): 221 labeled political claims with source hyperlinks
Fake News Challenge (Pommerleau & Rao 2017): 50K headline-body pairs, stance detection task
FEVER (Thorne et al. 2018): 185K claims requiring Wikipedia evidence retrieval and sentence selection
Wang dataset (2017): 12.8K claims from POLITIFACT with speaker metadata
HeroX challenge (Francis & Full Fact 2016): 90 labeled claims requiring manual verification

Surveys show that task formulations vary widely: some require binary labels, others multi-point scales; some provide evidence, others don't; some focus on political claims, others on numerical properties or entity relations.

Notes

Strengths:

First systematic unification of fact checking terminology and task formulations across disciplines, providing essential clarity to a fragmented literature
Clear taxonomy of evidence sources (text, structured knowledge, social networks, metadata) and their tradeoffs
Distinguishes between what computational systems actually output (labels, scores) versus what journalistic fact checking requires (justified verdicts)
Identifies evidence justification as a critical but neglected research direction

Weaknesses:

Focuses narrowly on NLP/database/ML approaches; social-science or psychological perspectives on misinformation are out of scope
Limited to English-language and text-based evidence; multilingual and multi-modal extensions barely discussed
Knowledge graph evidence section assumes closed-world assumption (all relevant facts are in the graph), which rarely holds in practice
Doesn't deeply engage with the political economy of fact-checking platforms or incentives that drive coverage decisions

Follow-ups:

FEVER (introduced by same authors, 2018) operationalized many recommendations about evidence justification
Subsequent work on evidence generation and natural language rationales has partly addressed the justification gap
Multi-modal claim verification (images, video) remains largely open despite being highlighted as important