Skip to content

Audio Forensics

Audio forensics encompasses technical methods for analyzing speech and audio recordings to authenticate their source, detect synthesis and manipulation, identify recording conditions, and verify speaker identity. The field bridges signal processing, acoustic analysis, and machine learning.

Core forensic questions

Authenticity: Is a recording genuine (human voice) or synthetic (generated via TTS, neural vocoder, or voice conversion)?

Generator attribution: If synthetic, which system produced it? Identification of the specific generator, acoustic model, and synthesis pipeline.

Recording conditions: What are the acoustic characteristics of the recording environment? Detection of codec, compression, noise, reverberation, and other channel effects.

Source verification: Do two recordings originate from the same source (speaker, generator, microphone)? Matching of acoustic signatures or metadata.

Speaker verification: Is the speaker who they claim to be? Biometric verification via voiceprint or prosodic features.

Technical approaches

Spectral analysis: Examining frequency content via Fourier analysis, spectrograms, or mel-frequency cepstral coefficients (MFCCs) to identify synthesis artifacts or environmental signatures.

Self-supervised learning (SSL): Frozen audio encoders trained on large speech corpora (Wav2Vec, HuBERT, Qwen3-ASR) learn representations that capture speaker and acoustic properties without labeled data.

Neural language models for audio: Large audio language models can score the likelihood of observed audio under different hypotheses, enabling forensic question-answering formulations.

Audio-visual synchronization: Cross-modal analysis of lip movements and speech to detect lip-sync mismatches or speech-to-video misalignment.

Adversarial robustness analysis: Evaluating whether detectors withstand intentional perturbations (background noise, codec compression, adversarial examples).

Key challenges

Generalization across generators: Detection trained on one voice synthesis architecture often fails on novel generators unseen during training.

Robustness to compression: Codec artifacts and social-media re-encoding significantly degrade detection performance.

Unknown unknowns: The most dangerous synthetic audio uses generators not present in training data; detection must handle open-set scenarios.

Faithfulness of explanations: When a forensic system makes a decision, its rationale must be verifiable against the audio itself, not hallucinated.

Computational efficiency: Real-time or near-real-time forensic analysis on edge devices or at platform scale requires efficient models.

Key papers in this wiki

  • VeriSpoof: Beyond Real or Fake with One Audio LLM — Unified framework answering four forensic questions (authenticity, generator attribution, recording conditions, source verification) using a single audio LLM; achieves 2.14% EER on Speech-DF-Arena; verifiable rationales generated for each decision.
  • CLRM: Cross-Layer Relation Modeling for Audio Deepfake Detection — Cross-layer relation modeling demonstrating that interactions between SSL encoder layers improve audio deepfake detection; 7.8–16.7% EER reduction on external benchmarks.
  • Detector-Guided Vocoder Training for Adversarial Text-to-Speech Synthesis — Demonstrates robustness vulnerabilities in audio forensics: integrating detector loss into vocoder training allows synthesis models to evade state-of-the-art detectors (RawNet2 EER 5.8%→39%, TSSD EER 3.4%→38%, AASIST EER 0.85%→21%); multi-detector strategy improves generalization to unseen architectures.

Open challenges

  • How can detection methods maintain high accuracy on generators with access to training data only at inference time?
  • What role do human listeners play in the forensic pipeline—should automated detection be paired with expert auditors?
  • How do we scale efficient audio forensics to platforms processing terabytes of audio daily?
  • Can forensic rationales be made sufficiently interpretable for use in legal proceedings or policy enforcement?