Contributions
- Epistemological critique of fact-checking methodology: Identifies systematic gaps between the philosophical requirements of factual knowledge and actual fact-checking practices in journalism.
- Selection effects: Demonstrates that fact checkers' mandatory choice of which claims to verify introduces political bias into coverage and candidate assessment.
- Compound fact problem: Shows that bundling multiple factual claims into single ratings, or disaggregating complex statements, can produce systematically biased verdicts.
- Causal inference mismatch: Argues that fact checkers lack the methodological sophistication of social science for evaluating causal claims; instead relying on informal "looking on the answer" comparisons.
- Future claims: Highlights the logical contradiction of rating unprovable predictions as true or false while claiming to verify facts.
- Definitional disputes: Demonstrates that fact checkers often adjudicate between competing definitions rather than checking facts—making them political arbiters, not neutral verifiers.
Method
The paper conducts a critical analysis of fact-checking practices using examples from PolitiFact, FactCheck.org, the Washington Post's Fact Checker, and other major outlets during the 2012 U.S. presidential election cycle. The authors analyze 1,057 fact-check articles and identify recurring patterns in methodological problems, using concrete cases (e.g., Romney's claims about oil production, Santorum's poverty statistics, Marina Navratilova's employment-discrimination claim) to illustrate each epistemological failure. The critique applies philosophical and social-scientific epistemology to journalism practice.
Results
The authors document systematic problems across multiple fact-checking outlets:
- ~40% of fact checks in their sample combined multiple claims into single ratings
- PolitiFact's summary ratings of 2012 candidates showed selection bias: Republican candidates received more "Pinocchio" ratings despite similar fact-accuracy patterns as Democrats
- Multiple fact-checking organizations rated the same claims differently, indicating lack of objective standards
- Fact checkers frequently rate causal claims and future predictions despite acknowledging these cannot be scientifically verified
- Fact checkers rarely articulate explicit criteria for evaluating evidence, keeping decision-making opaque
Notes
Strengths: This paper makes a rigorous, philosophical argument grounded in epistemology and social-science methodology. The examples are concrete and well-documented. The core insight—that fact-checking conflates objective verification with subjective judgment about definitions and causation—remains influential in critical media scholarship.
Limitations: The paper predates the expansion of fact-checking to other domains (health claims, COVID-19, AI-generated content). Some readers may view the critique as overly cynical about the possibility of objective fact-checking, though the authors explicitly acknowledge they agree facts exist but that journalism's tools are inadequate.
Impact: Cited by scholarship on fact-checking limitations, media criticism, and epistemology. The paper's arguments inform ongoing debates about fact-checker bias, the limits of binary true/false ratings, and whether fact-checking can be neutral journalism.