Current state of automated fact-checking
Vlachos emphasizes that deployed automated fact-checking technology is "nowhere near" trustworthy for full automation, particularly in contexts where errors carry life-or-death consequences. During the COVID-19 pandemic, despite increased urgency around medical fact-checking, he remained skeptical of deploying fully autonomous systems because the underlying technology is fundamentally unreliable at the level of decision-making required for healthcare guidance.
Why healthcare claims differ from political claims: Vlachos distinguishes between fact-checking political claims (e.g., "Is leaving the EU good or bad?") and scientific/medical claims (e.g., "Do vaccines cause autism?"). Political claims involve inherent subjectivity and value judgments—multiple defensible answers exist depending on priorities. Scientific claims have a factual basis: medical and scientific communities converge on evidence-based answers. This makes healthcare fact-checking a more tractable automation problem than political fact-checking, yet Vlachos remains cautious because confident-but-wrong automated decisions have catastrophic implications.
What automated fact-checking can do today
1. Claim matching to known fact-checks. When a new claim is a variant or repetition of a previously verified claim, automated systems can efficiently retrieve the existing fact-check without redundant human effort. Vlachos cites vaccination hoaxes as an example: the same misinformation circulates repeatedly with minor variations; matching systems can identify that "we already answered this" and resurface the prior fact-check, preventing duplicative work.
2. Consistency monitoring across languages. Vlachos describes a European research project (deutsche welle, a multilingual broadcaster) where automated systems helped ensure fact-checking consistency across language editions—e.g., ensuring the German, English, and French editions of the same story shared consistent fact-check verdicts. This addresses a real operational need for large media organizations without requiring full automation.
3. Temporal updates. Information evolves. Early pandemic guidance on masks changed as evidence accumulated. Automated systems can monitor knowledge bases and alert humans when new information conflicts with stored fact-checks, triggering human review to decide whether verdicts need updating. This addresses the problem of outdated fact-checks being "weaponized" by sharing them out of date but not marked as such.
Fundamental limitations of current methods
Wikipedia dependency. Most current automated fact-checking research operates on Wikipedia as the knowledge base. While Wikipedia is a valuable resource, it is not the only (or sometimes the best) source for verification, particularly for breaking news where Wikipedia lags behind reporting. Additionally, Wikipedia itself has editorial policies and biases; treating it as ground truth introduces those constraints into the system.
Simple claims only. Current methods handle narrow claim types: numerical claims verifiable against tables, simple factual statements that can be matched to sentences in a knowledge base. Complex, multi-part claims; claims about causality, counterfactuals, or systemic phenomena; claims requiring reasoning across multiple sources—these remain beyond current capabilities.
The role of human decisions. Even when automated systems identify relevant evidence, the final decision to label a claim true or false carries ethical weight that Vlachos believes should remain with humans. Returning a binary label ("true"/"false") oversimplifies complex claims. The system should instead present evidence and reasoning, allowing humans to make the final judgment.
From automated classification to human-AI collaboration
Vlachos critiques the framing of fact-checking as a classification problem (predict true/false). This perspective treats the task as analogous to sentiment analysis or spam detection—static labels applied to isolated inputs. But fact-checking is fundamentally about dialogue: engaging someone with a claim, understanding why they hold it, and presenting relevant information to help them reconsider.
The intellectual Turing test. Vlachos invokes the intellectual Turing test: the measure of whether a system can defend a viewpoint as well as someone who believes it. This is relevant to fact-checking because understanding a viewpoint requires understanding its logical structure, the evidence its proponents cite, and how they prioritize values. A fact-checker must be able to articulate why someone believes something before they can effectively counter it.
Dialogue-based fact-checking. Rather than a system that returns "this claim is false," Vlachos proposes a conversational agent that:
- Presents relevant information or sources the user may trust
- Asks probing questions ("Have you considered…?")
- Exposes the user to alternative perspectives without attacking them
- Encourages the user to think through the implications of their beliefs
This is not a recommendation system (which optimizes for engagement or click-through rates); it is a system optimized for expanding the user's understanding of a topic while respecting their autonomy and values.
Challenges in interactive fact-checking
Measurement complexity. Testing whether a dialogue-based system expands understanding is harder than testing whether it correctly classifies claims. It requires assessing whether users' views change, whether they better understand opposing viewpoints, and whether those changes persist—all of which requires post-intervention measurement and is more expensive than benchmark evaluation.
Balancing information and persuasion. Vlachos acknowledges a fine line between informing a user about diverse opinions and actively persuading them to adopt a particular viewpoint. A responsible system should expose users to viewpoint diversity without manipulating them toward a preferred conclusion. This is technically and ethically complex, especially given the history of AI systems being repurposed for propaganda.
Platform incentives. Fact-checking bots deployed on social media face misaligned incentives. Platforms optimize for engagement and time-on-site, while a dialogue-based fact-checker optimized for understanding might not maximize those metrics. Institutionally embedding such systems requires organizational buy-in from platforms, fact-checking organizations, or media outlets—not just algorithm design.
Practical examples and case studies
De-amplifying repeated falsehoods. Vlachos notes that when a false claim becomes widely recognized and repeatedly checked, automated systems can be confident flagging subsequent instances without re-triggering human fact-checkers. He contrasts this with novel falsehoods, where confidence must be lower and human judgment essential.
Medical claim evolution during COVID-19. The pandemic illustrated both the opportunity and the peril. On ventilators: initially believed to be protective, later evidence showed harms for some patients, changing the guidance. A fact-check database that updated guidance automatically based on new evidence would help; a system that maintained outdated guidance as truth would cause harm. This requires human-in-the-loop design.
Comparing fact-checking to other NLP tasks. Vlachos positions fact-checking as a structured prediction problem rather than classification. The output is not just a label but a justification: the evidence retrieved, the reasoning applied, the alternative interpretations considered. This is closer to question-answering or dialogue than to classification and should be evaluated accordingly.
Notes
This talk is valuable for its candid assessment of automated fact-checking limitations—particularly the honest acknowledgment that current technology is not ready for full deployment in high-stakes domains. Vlachos avoids both technological utopianism (treating fact-checking as a solved classification problem) and pessimism (declaring it impossible). Instead, he maps the space of feasible applications today and articulates a research agenda toward dialogue-based systems that engage users' reasoning rather than simply filtering or labeling content.
A strength is his emphasis on the distinction between political and scientific claims, which clarifies scope for fact-checking research—not all domains are equally amenable to automation. His critique of fact-checking-as-classification reflects broader NLP methodological debates: tasks involving high-stakes decisions, reasoning over complex evidence, or human behavior change may not be well-modeled as supervised learning on fixed datasets.
The dialogue-based perspective is underdeveloped in the fact-checking literature and worth pursuing. However, Vlachos acknowledges the research is nascent and practical deployment faces significant institutional and technical barriers. His cautious framing—"give me a few years"—is appropriate given the complexity of the problem.