Skip to content
Detector-Guided Vocoder Training for Adversarial Text-to-Speech Synthesis

Detector-Guided Vocoder Training for Adversarial Text-to-Speech Synthesis

Authors: Jiayan Cui, Davide Salvi, Paolo Bestagini, Stefano Tubaro

Venue: arXiv cs.SD, November 2026 — arXiv:2611.04511

TL;DR

This paper presents a method to generate synthetic speech that evades deepfake detectors by integrating detector loss directly into vocoder training, rather than applying post-hoc perturbations. The approach fine-tunes a BigVGAN vocoder while freezing the acoustic model and detector, generating adversarial speech that retains high perceptual quality and generalizes across multiple detector architectures. Experiments show the method reduces detector equal error rates by ~35–40 percentage points and paradoxically improves speech naturalness.

Contributions

  • Detector-guided adversarial synthesis: Proposes a novel TTS framework that integrates frozen detector models into vocoder optimization to generate intrinsically adversarial speech without post-processing.
  • Real-time applicability: Adversarial fine-tuning occurs offline; inference has no additional computational overhead compared to standard TTS.
  • Multi-detector generalization: Demonstrates white-box, black-box, and multi-detector attack scenarios, with joint optimization against multiple detectors improving transferability to unseen architectures.
  • Enhanced speech quality: Adversarial fine-tuning unexpectedly improves both objective (Fréchet Audio Distance) and subjective (MUSHRA listening test) measures of speech naturalness.

Method

The paper proposes an adversarial TTS framework with two cascade modules:

Architecture: - Acoustic model (FastSpeech2 variant with speaker adaptation): text → mel-spectrogram - Vocoder (BigVGAN): mel-spectrogram → waveform

The acoustic model remains frozen; only the vocoder parameters are optimized. The composite loss combines the original vocoder objective \(L_V\) with adversarial loss \(L_D\) from one or more detectors:

\[L_{\text{total}} = L_V + \lambda L_D\]

where the detection loss encourages the vocoder to generate waveforms classified as authentic:

\[L_D = \mathbb{E}_t [- \log (1 - D (V(A(t))))]\]

Multi-detector extension: The framework extends to multiple target detectors via a weighted combination:

\[L_D = \sum_{k=1}^{K} \alpha_k \mathbb{E}_t [- \log (1 - D_k (V(A(t))))]\]

During optimization, detector parameters remain frozen and gradients are backpropagated only to update the vocoder. This preserves linguistic and speaker representations learned by the acoustic model while allowing the waveform generation to suppress detector-relevant forensic cues.

Results

Detection Performance (Equal Error Rate):

Against RawNet2 (white-box target): - Baseline vocoder \(V\): 5.8% EER - Single-detector attack \(V_R\): 39.05% EER - Multi-detector attack \(V_{RT}\) (RawNet2 + TSSD): 43.87% EER

Against TSSD (white-box target): - Baseline: 3.40% EER - \(V_R\): 38.33% EER - \(V_{RT}\): 43.31% EER

Against AASIST (black-box/unseen): - Baseline: 0.85% EER - \(V_R\): 21.09% EER - \(V_{RT}\): 23.26% EER

The multi-detector strategy improves transferability to unseen detectors, suggesting that jointly optimizing against multiple architectures suppresses shared forensic cues rather than overfitting to specific models.

Speech Quality: - Fréchet Audio Distance (FAD) improved consistently: 31.25 → 22.95 → 17.81 (~25% improvement per stage), indicating closer distribution to authentic speech. - MUSHRA perceptual listening test (20 expert listeners): Fine-tuned vocoders \(V_R\) and \(V_{RT}\) perceived as significantly more realistic than baseline \(V\), progressively approaching real audio quality.

Confusion matrix (RawNet2 vs. \(V_{RT}\)): True Negative Rate 0.946, True Positive Rate 0.073 — synthetic tracks consistently misclassified as real.

Connections

  • Related to Deepfake Detection via adversarial robustness evaluation against state-of-the-art detectors (RawNet2, TSSD, AASIST).
  • Extends Adversarial Attacks on Misinformation Detection methodology from computer vision/NLP to audio synthesis; integrates perturbation into generation rather than post-processing.
  • Contributes to anti-forensics research by demonstrating vocoder-level manipulation as an alternative to additive noise attacks.
  • Relevant to Synthetic Media Detection robustness: highlights the vulnerability of learned detectors to adversarial fine-tuning of synthesis models.
  • Methodologically related to text-to-speech (TTS) quality research (perceptual improvements via detector loss).

Notes

Strengths: - Novel framing of adversarial attack as synthesis-time optimization rather than post-hoc perturbation, eliminating inference overhead and reducing artifact exposure. - Thorough evaluation across white-box (RawNet2, TSSD) and black-box (AASIST) scenarios, with clear ablations isolating single vs. multi-detector effects. - Counterintuitive finding that adversarial fine-tuning improves perceptual quality (supported by FAD and listening test) suggests that detector-relevant forensic artifacts are also perceptually undesirable. - Code and audio samples are publicly available, supporting reproducibility.

Limitations: - Evaluation limited to single TTS architecture (FastSpeech2 + BigVGAN); generalization to other acoustic models or vocoders not tested. - Detector training uses a single internal dataset; transferability to detectors trained on radically different data (e.g., different synthesis engines, languages, acoustic conditions) remains unexplored. - No discussion of defenses or detector robustification beyond noting the arms race. - The paper frames this as defensive research but explicitly acknowledges potential for misuse in generating convincing synthetic speech.

Open questions: - Why does suppressing detector-relevant forensic cues simultaneously reduce perceptual artifacts? Investigation into the relationship between detector robustness and synthesis quality could inform both fields. - Does the multi-detector strategy generalize to detectors with fundamentally different architectures (e.g., self-supervised models, phonetic-level detectors)? - How does robustness degrade with multiple rounds of detector updates (i.e., if detectors are retrained against adversarially synthesized speech)?