Skip to content
To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech

To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech

Authors: Debajyoti Mazumder, Mamta, Abhirama Subramanyam Penamakuri

Affiliations: IISER Bhopal, King's College London, MBZUAI

Venue: arXiv cs.CL, 2026 — arXiv:2609.30227

TL;DR

Large Audio Language Models (LALMs) exhibit a significant text-speech modality gap: they verify written claims reliably but fail on spoken ones, even from the same model. The paper introduces VeriSpeak, a benchmark of 3,879 spoken claims with temporal, geographical, and relational facts, and demonstrates that retrieval-augmented LALMs reaching 86.1% accuracy require both speech understanding and grounded reasoning over retrieved evidence.

Contributions

  • VeriSpeak benchmark: A controlled probe set of 3,879 spoken claims spanning three fact categories (year, location, relation) with balanced true/false labels and demographic diversity (659 subject entities from 15 countries).
  • Multimodal fact verification evaluation: Systematic comparison of six settings across three model families (Qwen, Qwen2.5, Phi) showing that factual verification ability observed in text does not reliably transfer to speech.
  • Analysis of the text-speech modality gap: Decomposition of the gap into two components: the backbone's text-mode difference and the speech interface degradation, revealing that the speech interface itself is a major source of factual-knowledge accessibility problems.
  • Retrieval-augmented speech verification: Demonstration that transcript-based RAG with chain-of-thought reasoning significantly improves speech fact verification but transcript-free audio-query RAG provides only modest gains.

Method

VeriSpeak is constructed from a Wikipedia-enriched knowledge base of 659 celebrity biographies. The benchmark creates three fact categories requiring qualitatively different knowledge types: (i) temporal facts (years and dates)—straightforward numerical retrieval; (ii) geographical facts (locations, nationalities)—entity and spatial reasoning; and (iii) relational facts (employment, kinship, organizational affiliations)—structured relationship knowledge.

For each fact, the authors synthesize claims using single-speaker consistent TTS (Coqui) to isolate speech modality effects. Negative claims are generated through deterministic category-specific perturbations (year offsets, entity swaps, relation swaps) to ensure plausible but incorrect counterparts.

The evaluation compares five LALMs across three model families under controlled conditions: - Text-only (C0): Text claim, text-only LLM baseline - Speech-only (C2): Speech claim without retrieved evidence - Speech-only with CoT (C3): Speech claim with chain-of-thought prompting (no evidence) - Transcript-RAG (C4): Speech claim with retrieved evidence, using ASR transcript for retrieval - Transcript-RAG with CoT (C5): Speech claim with evidence and explicit reasoning

Additional retrieval variants test audio-query RAG (using CLAP embeddings) and text-RAG for speech-only conditions.

Results

Text-speech modality gap: Qwen-7B achieves 75.3% accuracy on text but only 50.4% on speech (24.9-point gap). Audio-Flamingo-3 drops from 71.8% to 58.5% (13.3 points). All model families exhibit consistent degradation across fact categories.

Retrieval effectiveness: Transcript-based RAG improves speech-only performance: Qwen-Audio-Chat from 50.4% to 51.8%, Qwen2-Audio-7B from 49.2% to 50.9%, Audio-Flamingo-3 from 58.5% to 60.0%. However, gains are modest (1.4–2.3 points on average).

Chain-of-thought reasoning: CoT with retrieval substantially improves performance. Transcript-RAG with CoT reaches 86.1% for Audio-Flamingo-next-think, a 19.7-point improvement over speech-only. CoT alone (without evidence) shows minimal gains or degradation.

Gap decomposition: The modality gap is not uniformly explained by text-mode LALM differences or speech-interface degradation alone. Speech interface failure to reliably relay factual knowledge—exacerbated by speech-to-claim signal loss during claim verification—is a primary contributor.

Retrieval-evidence shift: RAG often shifts verification focus away from the spoken claim toward the retrieved passage; 65% of retrieval-verified claims match retrieved evidence, while only 31% preserve the original claim as the verification target.

Connections

  • Multimodal fake news detection — extends fact-checking to cross-modal claims; speech-text modality gap parallels challenges in visual-text misinformation detection.
  • Fact-checking and corrections — foundational evidence-based claim verification paradigm that motivated this speech-specific work.
  • Retrieval-Augmented Generation — technique demonstrated to partially bridge text-speech gap through external evidence.
  • Large Language Models — LALMs are the primary technology for speech-based fact verification; understanding modality gaps across language models is critical.

Notes

Strengths: The benchmark's careful construction ensures controlled evaluation—synthetic speech from a single speaker isolates modality effects from speaker variation; deterministic negatives prevent surface-level heuristics. The systematic decomposition of the gap into backbone and interface contributions is methodologically clean. The finding that explicit reasoning (CoT) is necessary for leveraging evidence in speech settings has actionable implications for improving speech fact-checking systems.

Limitations: The benchmark focuses on a narrow, curated knowledge base (celebrity facts) with synthetic single-speaker audio. Real-world speech fact-checking would need to handle diverse speakers, accents, background noise, and unconstrained claim domains beyond biographical facts. The paper does not investigate whether the observed gaps persist on naturally-spoken claims or on factual domains beyond temporal/geographical/relational assertions.

Open questions: Can the text-speech gap be closed via better speech encoders or multimodal pretraining? Does the gap persist on naturally-spoken claims and open-domain facts? Can models be trained to better preserve the spoken claim as the verification target when retrieving evidence?