Skip to content

Challenges with Large Language Models: Implications for Fake News, Fact-Checking, and Misinformation Detection

Speaker: Preslav Nakov
Affiliation: Muhammad bin Zayed University
Year: 2026
Video ID: wEPusrz5R6A

TL;DR

Nakov surveys the twin challenges posed by large language models (LLMs) to misinformation detection and fact-checking: (1) LLMs generate coherent, persuasive, and factually-unreliable text—hallucinations are pervasive and often hidden; (2) LLMs can be weaponized to automate misinformation production at scale through paraphrasing, persona generation, and coordinated fake-news websites. He discusses technical defenses (fact-checking the model's output via claim decomposition and evidence retrieval, detecting machine-generated content via perturbation analysis, synthetic data generation for training robust detectors) and broader approaches (regulation, media literacy, understanding model limitations). The talk emphasizes that this is not a problem of individual fact-checking capacity—it is a structural problem requiring parallel solutions across content safety, detection, and societal resilience.

Key challenges

LLM hallucinations and their characteristics: Large language models produce text that sounds coherent, authoritative, and appealing but frequently contains false information. Hallucinations differ from other LLM errors: a translation error might omit information, but a hallucination adds false information not present in the source. Models speak in first person, use confidently authoritative tone, and anthropomorphize their outputs—all of which make users mistake them for reliable sources. The halo effect (if good at X, people assume competence in Y) applies directly: users who experience LLMs generating correct information in one domain may falsely trust them in domains where the model is mediocre or unreliable. Additionally, models are trained on fixed data and degrade over time as the world changes, yet they do not cite sources and cannot be easily corrected.

Automated misinformation at scale: Because LLMs can paraphrase claims in infinite variations, bad actors can use them to evade frequency-based detection that fact-checkers rely on. Fact-checking organizations have very limited capacity—checking one claim can take days or weeks—and they prioritize by frequency. If an actor uses an LLM to generate hundreds of paraphrases of the same false claim across social media, fact-checkers cannot keep up. This is already happening: ChatGPT-generated fake profiles exist on Twitter; fully automatic fake-news websites are deployed; coordinated networks of such sites amplify disinformation. The problem scales with model capability and accessibility.

Weaponization through persona and source generation: An attacker can take text from social media or emails, have an LLM personify the author (changing pronouns, voice, tone), and regenerate plausible fake statements attributed to real people. More dangerously, an actor can generate an entire website backing up a false claim—providing the appearance of an independent source—which then becomes "evidence" that LLMs cite in future generations. Overcoming this requires multiple sources and source vetting, which neither LLMs nor casual readers typically do.

Proposed technical solutions

Fact-checking LLM output via claim decomposition and retrieval: Rather than asking an LLM "Is X true?" and trusting its answer, decompose output into atomic claims, generate questions about each claim, retrieve external evidence (Wikipedia, web search, knowledge graphs), and ask the LLM to reason over the evidence to detect contradictions. This shifts the burden from the model's internal knowledge to external grounding. The approach also allows for generating natural-language explanations of why a claim is false or unverifiable.

Program synthesis for complex fact-checking: For claims requiring multi-step reasoning (e.g., "James Cameron directed Titanic, so if James Cameron was born in Canada, did a Canadian direct Titanic?"), use an LLM to generate a Python program that describes the reasoning step-by-step. Provide in-context examples and let the LLM generate a program, which is then executed against a knowledge base. This deterministic approach avoids LLM-generated reasoning errors and leverages symbolic logic.

Detecting machine-generated content: Perturbation-based detection works in white-box settings: replace random words with alternatives from the model's n-best list thousands of times, and observe perplexity changes. Machine-generated text shows consistent perplexity decrease under perturbation (because the model is making locally optimal choices), while human text shows random variation. In black-box settings, use datasets like M4 (multiple generators, domains, languages) to train detectors robust across different LLM architectures. Key finding: training and testing on the same domain and generator yields ~100% accuracy, but cross-domain and cross-generator performance degrades significantly—suggesting strong domain/model specificity.

Synthetic data generation for training robust fake-news detectors: Generate synthetic fake news using LLMs, but incorporate propaganda techniques (emotional language, false citations, authority appeals) intentionally. Training detectors on such data improves detection of human-generated fake news—demonstrating that propaganda techniques and generation method interact. This approach also explores the "future state" in which machine-generated fake news may become prevalent.

Claim normalization and multilingual support: Normalize noisy claims (e.g., from tweets) into structured forms that fact-checkers can check. Use LLMs with in-context learning and few-shot examples to normalize claims across multiple languages. This work spans English and multilingual setups (Asian languages, European languages, Arabic, etc.), expanding the capacity of fact-checking infrastructure.

Broader approaches

Safety mechanisms in LLMs: Incorporate safety constraints at multiple layers: in prompting (instruct the model to be factual), inside the model (instruction-tuning with safety objectives), in external classifiers (detect harmful outputs post-hoc), and in user-facing interfaces (warning labels, output filtering). Nakov's team built Jais, an Arabic-centric LLM, and found that long, repetitive system prompts were the most effective safety mechanism—more so than other technical interventions—suggesting that reinforced instruction is critical.

Regulation and open-source alternatives: GDPR, the EU AI Act, executive orders, and international declarations (e.g., the Beijing Declaration by 28 countries) are beginning to constrain LLM training and deployment. Regulation is necessary but risky—if overregulation stifles innovation, only bad actors (with no compliance constraints) will continue development. Open-source models are a positive counterbalance: they enable research, transparency, and alternatives to closed commercial systems, reducing monopolistic control and enabling the broader community to study and improve safety.

Media literacy and societal resilience: Teaching users to understand what LLMs can and cannot do—how they are vulnerable to prompt injection, how they hallucinate, how to cross-reference claims—can reduce credulous acceptance of generated content. An analogy to spam: 15-20 years ago, spam was a massive problem; today, it is controlled but not eradicated. The same trajectory is possible for misinformation: control through a combination of technical detection, source authentication, media literacy, and societal adaptation—not elimination.

Propaganda detection and framing analysis: Nakov's team analyzes propaganda techniques (appeal to fear, loaded language, appeal to authority, etc.) across multiple languages. Work includes detecting techniques in text spans, analyzing framing (how an issue is discussed: political, economic, health, moral lens), and applying these tools to large corpora (2+ million articles on the Russia-Ukraine conflict). This work bridges misinformation detection and broader propaganda studies.

Key research efforts

  • Propaganda detection: SemEval shared tasks on propaganda and persuasion detection; cross-lingual annotated datasets covering French, Polish, English, Russian, Italian, German, Greek, Georgian, Spanish, Portuguese, Chinese, Bulgarian, Arabic, Macedonian.
  • Fact-checking with LLMs: Claim decomposition, evidence retrieval, question-answering for fact-checking.
  • Machine-generated text detection: Perturbation-based detection in white-box settings; M4 dataset for black-box detection across multiple generators, domains, and languages.
  • Jais language model: 13 billion and 70 billion parameter Arabic-centric LLM with strong multilingual capability; incorporates safety mechanisms.
  • LLM 360 initiative: Fully open, fully transparent LLM training—every experiment, checkpoint, input-output pair, and evaluation is released; current focus on code and text generation.

Connections

Notes

This talk is notable for its comprehensive treatment of LLMs as a dual problem: (1) LLMs themselves are unreliable sources of factual information, requiring new approaches to ground their outputs in evidence; (2) LLMs are tools for producing misinformation at scale, requiring detection and attribution work. The talk integrates technical solutions (perturbation-based detection, claim decomposition, program synthesis) with regulatory and social approaches (media literacy, open-source alternatives, international cooperation).

Nakov's emphasis on perturbation-based detection deserves highlight: the method is simple, interpretable, and requires only the ability to sample from the model—no access to internal parameters. The finding that white-box detection is near-perfect while cross-domain performance degrades is sobering, suggesting detection must be domain-specific and continuously updated as attacks evolve.

The distinction between misinformation (false information spread unintentionally) and disinformation (deliberately false information) is important throughout, as are the terminological shifts from "fake news" (which sounds like false content only) to "information manipulation" or "information disorder" (which captures weaponization and propaganda). The WHO's coining of "infodemic" during the pandemic underscores this: the core problem is not falsity alone, but harmful information spread.

The talk also stresses that LLM hallucination is not purely a technical problem—it is fundamentally a societal problem. Even if hallucinations were eliminated, malicious actors could still use LLMs to generate targeted misinformation. And even if misinformation generation were prevented, users who do not understand LLM limitations would fall for coherent but unreliable outputs. This multi-layered nature means no single technical fix suffices.