Skip to content

Fact Checking: Task definition and dataset construction

Authors: Andreas Vlachos, Sebastian Riedel

Venue: ACL 2014 Workshop on Language Technologies and Computational Social Science, Baltimore, Maryland, June 26, 2014 — pp. 18–22

TL;DR

This paper introduces fact-checking as a formal computational task: assessing the truthfulness of political claims as an ordinal classification problem with five labels (TRUE, MOSTLYTRUE, HALFTRUE, MOSTLYFALSE, FALSE). The authors construct a dataset of 106 objectively-verifiable fact-checked statements from PolitiFact and Channel 4 (out of 221 examined), and discuss how the task maps to core NLP challenges including textual entailment, information extraction, and semantic parsing.

Contributions

  • Formal task definition: Defines fact-checking as ordinal classification of claims into five truth categories, emphasizing the role of context (speaker, time) and the challenge of handling multiple interpretations of data
  • Public dataset: 106 suitable fact-checked claims from Channel 4 Fact Check and PolitiFact, with verdicts, dates, speakers, and sources; selection criteria exclude causal attribution, future speculations, and opinion-based claims
  • Relevance criteria: Identifies statements that can be fact-checked objectively using verifiable sources and data, excluding statements whose truthfulness depends on undefined terms or personal interpretation
  • Baseline approaches: Proposes three strategies: (1) supervised ordinal classification on claims (unlikely to work without world knowledge), (2) k-nearest neighbor retrieval matching new claims to previously fact-checked ones, (3) matching claims against a curated text collection (e.g., Wikipedia)
  • Connections to NLP: Maps fact-checking stages (statement extraction, question construction, evidence retrieval, verdict synthesis) to established NLP tasks—hypothesis testing as speculation detection, evidence sourcing as information extraction, and verdict derivation as textual entailment

Method

Task definition: Fact-checking is defined as assigning a truth value to a claim made in a particular context. The paper uses ordinal classification (Frank & Hall, 2001) to capture nuanced verdicts rather than binary true/false labels. Context includes the speaker identity and the temporal frame—essential for assessing quantitative claims anchored to specific time periods.

The paper explicitly restricts scope to objectively checkable claims: those verifiable against factual sources (statistics, records, encyclopedic knowledge). It excludes causal attribution (linking statistics to specific laws), future speculation, opinion-based assertions, and statements with undefined terms that lack universal definitions.

Dataset construction: Authors surveyed popular fact-checking websites, focusing on Channel 4 Fact Check (UK) and PolitiFact (US) as sources with large archives (>1,000 statements each) and detailed verdicts (MOSTLYTRUE, HALFTRUE, etc.). For each statement, they recorded: the claim itself, date made, speaker, verdict label, and supporting URL.

They examined 221 recent fact-checks and identified 106 as suitable for the task based on objective verifiability criteria. The two websites use different labeling schemes; verdicts were aligned to a five-point scale: TRUE, MOSTLYTRUE, HALFTRUE, MOSTLYFALSE, FALSE.

For suitable claims, authors collected the sources (statistics tables, government reports, think tank analyses) cited in journalists' analyses. They excluded verdicts relying on unavailable online sources (personal communications) and noted that journalists' analyses sometimes exhibit language patterns correlated with their verdict, which could enable solving part of the task as sentiment analysis rather than factual reasoning.

Results

  • Dataset: 106 suitable claims from 221 examined (47.96% suitable rate)
  • Verdict distribution: Claims labeled across the five-point truth scale, reflecting journalist assessments of nuance (e.g., HALFTRUE for claims verifiable through multiple reasonable interpretations of data)
  • Challenge characterization: Demonstrates that fact-checking requires:
  • Relational extraction and interpretation (extracting facts from sources)
  • Temporal reasoning (anchoring claims to time-frames)
  • Multiple source consultation (combining evidence from diverse sources)
  • Data interpretation (recognizing valid alternative interpretations)

Connections

Notes

Foundational work that introduced the automatic fact-checking task to the NLP community and established the problem formulation that has dominated the field for a decade. The paper's emphasis on context (speaker, time) and objective verifiability remains underexplored in subsequent work focusing primarily on evidence matching and entailment. The ordinal classification framing (TRUE/MOSTLYTRUE/HALFTRUE/...) is widely adopted in modern fact-checking systems and benchmarks.

The limitation that journalists' analyses contain linguistic signals correlated with verdicts (e.g., hedging language, intensifiers) foreshadows later findings that sentiment analysis features can sometimes proxy for veracity prediction—a phenomenon that complicates interpretability in automated systems.

The paper identifies question construction as a novel task requiring semantic parsing and logical reformulation, which remains largely unsolved in subsequent work; most systems focus on evidence retrieval and entailment rather than interpretable question decomposition.