Fighting Misinformation with AI: Open, Safe, and Factual Language Models

Preslav Nakov

In brief

Nakov argues that fighting misinformation in the era of large language models requires three simultaneous efforts: (1) building open and transparent language models with full release of data, training experiments, and evaluation metrics; (2) implementing safety mechanisms at multiple levels (data cleaning, instruction tuning, model-level constraints, prompt engineering); (3) ensuring factuality through retrieval-augmented generation, fact-checking, uncertainty quantification, and user-facing tools like OpenFactCheck and LM Polyraph. He demonstrates tools for propaganda detection, machine-generated text detection, and discusses regulation and media literacy as necessary complements to technological solutions.

Key themes

1. Open and transparent language models

Truly open release: Beyond releasing model weights, true openness requires publishing all training data, evaluation metrics, and every experiment conducted during development. Nakov's team released 360 experiments alongside their models to enable reproducibility and research.

Language-specific models: Pretrained multilingual models perform poorly on low-resource languages because tokenization breaks words into many fragments. For example:

English "hello" → few tokens
Bulgarian "здравей" → breaks into many pieces, wasting context window and increasing compute 4×
Arabic → tokenizes almost to individual letters

This inefficiency has major consequences: cost quadruples for the same text length, context window shrinks, and semantic capacity is reduced. Solutions include post-vocabulary adaptation or training from scratch with optimized tokenization (e.g., ensuring 100 words tokenize to ~101 tokens rather than 400).

Models released by Nakov's team:

Jais (70B, 65B, 30B): Arabic-centric model, trained from scratch (not adapted), optimized for Arabic dialects. Available on Hugging Face and Microsoft Azure.
Nanda: Best-in-class Hindi language model; bilingual (Hindi + English).
Fat: Moroccan Arabic dialect model.
Vikuna: Early attempt to reproduce ChatGPT, collaborative across universities.
Multiple models for genome sequences, climate change, cardiovascular diseases, and code generation.

2. Safe language models

Multi-layered safety approach: Safety mechanisms must be embedded at multiple levels because any single level can fail:

Data cleaning (remove harmful content, personal information)
Instruction tuning (teach the model to refuse harmful tasks)
Model-level constraints (built-in refusals)
Deployed systems (detect and filter harmful outputs post-hoc)
Prompting (system prompt with safety instructions)

The Don't Answer dataset: A three-level hierarchy of harmful content:

Self-harm, depression, illegal activities, harassment, trolling
Hate speech
Misinformation, unsolicited advice (financial, medical, legal), personal information

This dataset has been adapted to multiple languages: Arabic, Chinese, German, Bulgarian, Hindi, Kazak (and more in progress).

System prompt as the strongest safety lever: Despite intuition pointing to data cleaning or instruction tuning, Nakov found that long, repetitive system prompts are the most effective safety mechanism—the model finds it extremely hard to override explicit instructions embedded in the prompt. This also prevents users from writing prompts that overpower the safeguards.

3. Factual language models

The hallucination problem: LLMs generate fluent, persuasive, coherent text that often contains false information. Hallucinations differ from other errors:

They add information not present in source material (unlike omissions)
They are often fluent and convincing (unlike garbled text)
Larger models hallucinate less because they have capacity to memorize; smaller models must be "creative" more often

Terminology note: Nakov dislikes the term "hallucination" because it misapplies psychology—the term originally meant generating information not in the source (which can be true or false). What LLMs actually do is generate text with uncertain factuality. A more precise distinction: models either "recite" what they have seen or are "creative"; hallucination occurs during creative mode.

Retrieval-augmented generation (RAG): Provide the model with external evidence before asking it to reason:

Decompose claims into atomic sub-claims
Generate questions about each claim
Retrieve evidence from Wikipedia, web, knowledge graphs
Have the model reason over evidence and detect contradictions
Generate natural-language explanations for why claims are false

Uncertainty quantification (LM-Polyraph): Rather than asking if a claim is true/false, look at the model's internal confidence distribution:

Certain vs. uncertain tokens can be highlighted (e.g., "20 Grammy prizes" vs. "awards"—stylistic, doesn't matter)
Some uncertainties propagate (e.g., "British" vs. "American" for Madonna conditions all downstream reasoning)
The method identifies which uncertainties actually matter for factuality

Machine-generated text detection:

Perturbation-based (white-box): Replace random words with alternatives from the model's n-best list thousands of times; machine-generated text shows consistent perplexity decrease, while human text shows random variation
Black-box (M4 dataset): Train on multiple generators, domains, and languages; cross-domain/cross-generator performance degrades significantly, showing detection is generator and domain-specific
Four categories: human-written, machine-generated, human-written-machine-polished, machine-written-human-polished

4. Propaganda and framing

Misinformation is not just about falsity—it's about weaponization. Propaganda uses specific techniques:

Emotional appeals (fear, anger)
Logical fallacies (appeal to authority, false dilemma)
Source-based manipulation (who is saying it?)

SemEval propaganda detection shared tasks have annotated corpora across multiple languages and developed systems to identify propaganda techniques in text spans.

Framing analysis: The same factual claim can be framed differently depending on the perspective:

COVID-19 from health, economic, legal, or moral perspective
Brexit or Russia-Ukraine conflict from different framings

Narrative extraction: Identify the specific narrative threads in longer texts and analyze their prevalence across countries and media.

5. Terminology: From "fake news" to "infodemics"

2016: "Fake news" (focus on falsity alone)
2018-2020: "Disinformation" and "misinformation" (focus on weaponization and harm)
COVID-19 era: "Infodemic" (coined by WHO; emphasizes harm regardless of truth value)

Nakov notes that the term shift reflects deepening understanding: the core problem is not falsity alone, but information that causes harm, whether true or false. Cherry-picked facts can be weaponized even if individually true.

Tools and systems

OpenFactCheck: A platform for evaluating and building fact-checking systems. Users can:

Evaluate any text for factuality
Evaluate an LLM's generation for reliability
Evaluate a fact-checking system for accuracy
Build custom fact-checking pipelines by selecting preprocessing, evidence retrieval, and reasoning components

The tool integrates multiple fact-checkers and shows how different pipeline choices affect results. Full code and methods are open-source.

LM-Polyraph: Analyzes internal model confidence to identify which parts of a generation are uncertain and which uncertainties actually matter for factuality. Provides highlighting of uncertain claims.

Propaganda detection system: Detects propaganda techniques in text (fear, name-calling, doubt-casting), supports multiple languages, and provides a database of analyzed media articles with framing analysis and persuasion metrics across multiple countries.

Machine-generated text detector: Classifies text into human-written, machine-generated, human-written-machine-polished, or machine-written-human-polished categories.

Institutional support and resources

Nakov's work is enabled by:

Mohamed bin Zayed University of Artificial Intelligence (MBZUAI): World's first AI-dedicated university, founded 2021, department of NLP founded 2023. Ranked 10th globally in AI research; his department is ranked 2nd globally despite being only 2.5 years old.
Compute resources: A supercomputer with 54 million cores, enabling training of 65-70B models in ~10 days (crucial for research at this scale).
Collaboration: Partnerships with fact-checking organizations (Full Fact UK, Newtral Spain, PolitiFact Italy), Meta researchers, and European Commission Joint Research Centre.

Broader implications and societal context

Three information revolutions:

Printing press: Made it possible for anyone to print (removed gatekeepers on production)
Social media: Made anyone a news agency (removed gatekeepers on distribution)
AI/LLMs: Anyone can write convincingly and persuasively at scale (removed gatekeepers on authorship)

In all three cases, old gatekeepers disappeared, new technologies democratized capabilities, the public was unprepared and unprotected, and governments were slow to respond.

Challenges with regulation:

Bad actors do not obey regulation; overregulation stifles research
Different geographies have different laws (e.g., content legal in one country may be illegal in another, requiring context-dependent safety mechanisms)
Temporal dependency: scientific consensus changes (e.g., cholesterol in eggs, COVID-19 transmission, vaccine development)
Explainability matters: even when a system gets the answer wrong, the explanation can provide value that a human evaluator recognizes

Media literacy and societal resilience:

Public understanding of LLM capabilities and limitations is critical (e.g., lawyers filing briefs with ChatGPT-generated citations, hallucinated case law)
Regulation should be minimal but strategic: GDPR, EU AI Act, and international declarations are beginning to constrain but should not block innovation
Open-source models provide a counterbalance to commercial closed systems, enabling research and alternatives
The trajectory resembles spam management: 15-20 years ago, spam was endemic; today, controlled but not eradicated. Misinformation may follow a similar path through parallel technical, regulatory, and societal adaptation

Key research efforts and datasets

Propaganda detection: SemEval shared tasks; cross-lingual annotated datasets (French, Polish, English, Russian, Italian, German, Greek, Georgian, Spanish, Portuguese, Chinese, Bulgarian, Arabic, Macedonian)
Jais and Nanda: Arabic-centric and Hindi-centric language models with strong multilingual capability
LLM 360 initiative: Fully transparent LLM training with all experiments, checkpoints, and data released
Machine-generated text detection: M4 dataset with multiple generators, domains, and languages
Fact-checking with LLMs: Claim decomposition, evidence retrieval, question-answering, uncertainty quantification

Notes

This keynote is valuable for its framing of the misinformation problem at the intersection of generation (LLMs as tools for producing misinformation) and detection (LLMs as unreliable sources requiring grounding). The emphasis on open, transparent, and language-specific models reflects a conviction that democratizing AI capability is preferable to concentrating it, and that English-only or multilingual-averaged models harm less-resourced languages.

The multi-layered safety approach is pragmatic: no single mechanism (data cleaning, instruction tuning, prompting, filtering) is sufficient; each can be circumvented, so defense-in-depth is necessary. The finding that system prompts are the strongest lever is particularly useful for practitioners.

The discussion of temporal and geographic variation in truth (cholesterol in eggs, COVID-19 masks, vaccine availability) is a crucial reminder that static fact-checking or regulation cannot work. Systems must be dynamic, context-aware, and willing to update as consensus evolves.

The terminology shift from "fake news" to "infodemic" captures an important reframing: harm matters more than falsity alone. A cherry-picked set of true facts used to mislead is as dangerous as a false claim. This perspective should inform evaluation: fact-checkers need to assess not just truth but weaponization and potential harm.

Finally, the mention of lawyer-filed briefs citing ChatGPT-hallucinated case law underscores that the misinformation problem is not just about bad actors—it is about well-intentioned people who do not yet understand LLM limitations using powerful tools naively. Media literacy for LLMs is therefore a public health priority.