Skip to content
VeriSpoof: Beyond Real or Fake with One Audio LLM

VeriSpoof: Beyond Real or Fake with One Audio LLM

Authors: Yang Xiao, Ting Dang
Affiliation: University of Melbourne
Publication: Technical Report, November 2026
ArXiv: 2611.04634

TL;DR

VeriSpoof is an audio language model system that unifies multiple forensic tasks in a single neural framework: detecting whether a speech recording is synthetic, identifying which generator produced it, analyzing recording conditions, and verifying whether two recordings share a source. The system achieves state-of-the-art results on detection (2.14% equal error rate on Speech-DF-Arena, ranked 3rd among 28 open-weight systems) while generating faithful, verifiable rationales for each decision using only 1% of trainable parameters.

Contributions

  • Unified framework addressing four forensic questions in speech deepfake analysis: detection, generator attribution, recording-condition analysis, and source verification using a single model with frozen encoders and LoRA-adapted LLM
  • Forensic question-answering formulation where answers are scored via audio token likelihoods, enabling threshold-based operation without separate classification heads
  • Grounded rationale generation with faithfulness verification through counterfactual training; statements are checkable against audio waveforms and labels
  • Analysis showing that learned audio representations, not just the LLM head, determine detection outcomes; encoder improvements directly impact forensic accuracy
  • Computationally efficient: only 29.7M of 2.85B LLM parameters are trained; training fits on 24GB GPU; inference requires ~8GB
  • Comprehensive benchmark and open-sourced models (VeriSpoof-Base and VeriSpoof-Plus) with evaluation suite covering detection, attribution, source verification, recording conditions, and grounded reasoning

Method

VeriSpoof reformulates speech forensics as a scored question-answering task. The system pipeline:

  1. Audio encoding: Incoming recordings are encoded via a frozen Qwen3-ASR encoder, then passed through learned layer mixing over the encoder's 25 layers to adaptively aggregate features
  2. Tokenization: An audio tokenizer pools the mixing output into 16 utterance tokens and stacks frame pairs for temporal context
  3. Question-answer scoring: The model receives a forensic question (e.g., "Is this recording genuine?") with candidate answers written as sentences. A frozen LLM with LoRA adapters reads the encoded audio and scores each answer via token likelihood: \(s_q(u) = \log p_\theta(a^+ | u, q) - \log p_\theta(a^- | u, q)\)
  4. Four forensic families: VeriSpoof handles authenticity (genuine vs. spoofed), generator attribution (which synthesis architecture), recording conditions (codec, noise, reverberation), and source verification (same generator for two recordings)
  5. Rationale generation with checks: For each decision, the model generates structured rationales whose statements are independently verified against audio observations and held-out labels, ensuring faithfulness without relying on unverified generated text
  6. Training stages: Three stages of fine-tuning—perception alignment, forensic question tuning, and grounded rationale tuning—with counterfactual examples teaching the model to revise conclusions when evidence is edited

Results

Detection: 2.14% equal error rate (EER) pooled across 14 test sets from Speech-DF-Arena (ranked 3rd of 28 open-weight systems). On same-source pairs from generators unseen in training, 6.2% EER; on known generators, 15.0% EER.

Generator attribution & recording conditions: The same weights identify acoustic model, vocoder, codec, and recording conditions. Handles both known and unknown architectures.

Faithfulness: 99.6% accuracy on PartialEdit (partial fakes caught after reasoning); 99.5% of decisions follow stated evidence. Counterfactual training allows the model to revise conclusions when audio is edited or evidence contradicted.

Efficiency: Only 1% of parameters trained. Full training with gradient checkpointing on 24GB GPU fits within budget; inference at ~2.24s per recording (PyTorch), accelerated to 46× speedup with batching via vLLM.

Connections

Notes

Strengths: - Unifies multiple forensic tasks that typically require separate models or pipelines, reducing engineering complexity - Faithful rationales are checked during training, making explanations verifiable rather than hallucinated - Computationally lightweight: LoRA fine-tuning of a frozen foundation model is practical for deployment - Handles unknown architectures (generators unseen during training), a key real-world requirement - Strong empirical results on public benchmarks with transparent comparison to 27 other open-weight systems

Limitations & open questions: - Generalization to novel synthesis methods not seen during training remains partially unexplored; 6.2% EER on new generators suggests room for improvement - The method relies on audio token likelihoods as a proxy for decision confidence; a deeper analysis of how well this correlates with actual forensic confidence would strengthen claims - Counterfactual training is limited to edits held out from training; robustness to other manipulations (e.g., speaker conversion, style transfer) not fully characterized - The paper reports results on Speech-DF-Arena but does not exhaustively compare to all concurrent audio deepfake detection systems; a few recent methods appear absent from benchmarks