ReplayFactor: Replay-Aware Dual-Attribute Learning for Practical Speech Deepfake Detection

Authors: Miao He, Jiayi Zhou, Yu Zheng, Zhongjie Ba, Peng Cheng, Yuxi Zhou, Junjie Zhu, Changhao Zhang, Jian Liu

Venue: arXiv cs.SD, 2026 — arXiv:2611.02628

TL;DR

Existing speech deepfake detectors fail on physically replayed synthetic speech (played through a loudspeaker and re-recorded), a low-cost, practical threat to voice authentication systems. ReplayFactor addresses this by learning both source authenticity (human vs. generated) and replay presence (non-replayed vs. replayed) via SSL-based representations fused with acoustic priors and supervised contrastive learning in dual embedding spaces. On EchoFake, the method improves macro-F1 from 60.99% to 76.44% and reduces EER from 11.86% to 3.05%; a new Chinese dataset (ReplayFake-CN, 185k utterances) supports evaluation across diverse synthesis methods and replay devices.

Contributions

• Dual-attribute learning framework: Explicitly factorizes speech authentication into source (bona fide vs. generated) and replay (non-replayed vs. replayed) attributes, enabling each to be learned independently while maintaining consistency across the other.
• Replay-aware feature fusion: Enhances SSL-based replay embeddings with Constant-Q Cepstral Coefficients (CQCCs) via gated fusion, capturing low-level replay-sensitive cues (high-frequency attenuation, channel distortions) that SSL alone misses.
• Multi-task and dual-space contrastive learning: Jointly optimizes 4-class main classification with two binary attribute tasks and supervised contrastive learning in source and replay embedding spaces, promoting cross-condition consistency (e.g., humans remain human-like whether replayed or not).
• ReplayFake-CN dataset: Large-scale Chinese audio deepfake dataset with 185k replayed synthetic utterances, 1,921 speakers, 22 TTS systems, 90 playback–recording device pairs, enabling evaluation of cross-domain generalization (unseen synthesis models, unseen devices, unseen combinations).
• Strong generalization: Substantial gains on public benchmarks despite closed-set performance trade-off; 38.31% and 40.32% F1 on ReplayDF (RB and RF), vs. 6.73% and 25.94% for baselines; 96.95% accuracy on unseen synthesis models in ReplayFake-CN.

Method

Problem formulation: Each utterance is assigned a 4-class label: {Bona fide, Fake, RB (Replayed Bona fide), RF (Replayed Fake)}. These are factorized into: - Source attribute \(y^s\): Human speech {B, RB} vs. generated speech {F, RF} - Replay attribute \(y^r\): Non-replayed speech {B, F} vs. replayed speech {RB, RF}

Component I: Replay-Aware Feature Fusion

A shared SSL representation is extracted using a pretrained SSL encoder (MMS-300M) followed by an AASIST graph-attention backend. Two attribute-specific branches map this into: - Source embedding \(e^s_i\): Discriminates human from generated speech - Replay embedding \(e^r_{i,ssl}\): Discriminates non-replayed from replayed speech

To strengthen replay detection, a complementary acoustic prior (Constant-Q Cepstral Coefficients) is extracted, computed as:

\[c_i = \text{DCTResample}\left(\log|CQT(x_i)|^2 + \epsilon\right)$$ $$q_i = [c_i, \Delta c_i, \Delta^2 c_i], \quad e^a_i = \text{LCNN}(q_i)\]

The acoustic embedding captures high-frequency differences and channel response distortions diagnostic of replay.

Gated fusion: The acoustic and SSL replay embeddings are combined via sample-wise gating:

\[e^r_i = e^r_{i,ssl} + g_i e^a_i\]

where \(g_i \in (0, 1)\) adaptively controls acoustic contribution per utterance. This residual design preserves SSL features while incorporating complementary acoustic cues.

Component II: Multi-Task and Dual-Space Contrastive Learning

Three classification objectives: - 4-class main task: \(L^4_{CE}\) on concatenated representation \([e^s_i; e^r_i]\) - Source auxiliary task: \(L^s_{CE}\) on \(e^s_i\) (binary: human vs. generated) - Replay auxiliary task: \(L^r_{CE}\) on \(e^r_i\) (binary: non-replayed vs. replayed)

Supervised contrastive learning in two spaces: - Source space: Positives group B↔RB (both human) and F↔RF (both generated), enforcing source consistency across replay states. - Replay space: Positives group B↔F (both non-replayed) and RB↔RF (both replayed), enforcing replay consistency across source types.

For each mini-batch \(B\) and anchor \(i\) in space \(u \in \{s, r\}\):

\[L^u_{sup} = -\frac{1}{N} \sum_{i=1}^{N} \frac{1}{|P^u_i|} \sum_{p \in P^u_i} \log \frac{\exp(s^u_{ip})}{\sum_{a \in B \setminus i} \exp(s^u_{ia})}\]

where \(s^u_{ij} = (z^u_i)^\top z^u_j / \tau^u\) is temperature-scaled cosine similarity.

Overall loss:

\[\mathcal{L} = L^4_{CE} + \alpha_s L^s_{CE} + \alpha_r L^r_{CE} + \lambda_s L^s_{sup} + \lambda_r L^r_{sup}\]

Results

EchoFake (public benchmark)

Open-set evaluation (most challenging; test set includes unseen TTS models): - Baseline (XLSR-Linear): macro-F1 60.99%, EER 11.86% - ReplayFactor: macro-F1 76.44% (+25.3%), EER 3.05% (−74%)

Per-class F1 scores on open set: - B (bona fide): 93.91% (+15.08) - RB (replayed bona fide): 61.12% (+20.99) — largest gain, most difficult class - F (fake): 82.01% (+6.37) - RF (replayed fake): 68.72% (+19.38)

Closed-set evaluation: Slightly lower performance (97.93% macro-F1 vs. 98.81% baseline) due to optimization for open-set robustness.

ReplayDF (cross-dataset)

Models trained on EchoFake tested on ReplayDF (English, different synthesis methods and devices): - Baseline (XLSR-Linear): F1 = 7.72% (RB), 19.02% (RF) - ReplayFactor: F1 = 38.31% (RB), 40.32% (RF)

Substantial gains but cross-dataset generalization remains challenging, highlighting domain shift from English to multilingual contexts and device variations.

ReplayFake-CN (new Chinese dataset)

Unseen synthesis models: 96.95% accuracy (vs. 93.97% baseline) Unseen devices: 83.00% accuracy (vs. 82.59% baseline) Unseen synthesis–device combinations: 73.75% accuracy (vs. 73.30% baseline)

Largest gains under unseen synthesis models; device-related domain shifts remain harder to overcome despite abundant training data (400k samples vs. 40k on EchoFake).

Ablation Study (EchoFake open set)

Configuration	EER	macro-F1
4-class baseline	7.26%	62.27%
w/o acoustic prior	8.11%	73.54%
w/o supervised contrastive learning	2.70%	73.89%
w/o 4-class head	4.31%	69.22%
Full model	3.05%	76.44%

• Acoustic prior removal increases EER to 8.11%, confirming low-level cues are essential for replay robustness.
• SupCon removal maintains low EER (2.70%) but reduces macro-F1 to 73.89%, mainly from RB class performance drop, suggesting it stabilizes source–replay consistency.
• 4-class head removal causes largest macro-F1 drop (69.22%), indicating main task captures source–replay interactions missing from attribute tasks alone.

Connections

• Deepfake Detection — broader deepfake detection framework
• Audio Deepfakes — audio synthesis and voice cloning detection
• Speech Anti-Spoofing — anti-spoofing focuses on replay attacks
• Synthetic Media Detection — general synthetic media detection methodology
• Contrastive learning for detection — supervised contrastive learning as learning paradigm
• CLRM: Cross-Layer Relation Modeling for Audio Deepfake Detection — complementary SSL-based audio deepfake approach using cross-layer modeling
• GraphSeal: Graph-Augmented Speech Deepfake Detection via Temporal-Spectral Artifact Propagation — graph-based temporal-spectral artifact propagation for detection
• Detector-Guided Vocoder Training for Adversarial Text-to-Speech Synthesis — adversarial TTS generation targeting deepfake detectors

Notes

This work makes replay attacks (physical playback-recording) central rather than auxiliary; most detection research assumes digitally generated audio and sees replay as a secondary robustness challenge. The insight that replay distortions are fundamentally different from synthesis artifacts, requiring explicit learning of both source and replay attributes, is well-motivated and practically important for deployed systems.

The dual-attribute design is elegant: factorization allows learning each aspect independently while contrastive learning ensures consistency (e.g., a human voice should remain acoustically human even after being replayed). Results validate this: the source and replay attribute spaces show clear separation (t-SNE visualization) with only minor RB/RF overlap in joint space.

Limitations: Cross-dataset generalization to ReplayDF remains weak (38–40% F1), especially for RF. The authors note that device-related domain shifts (even within the large ReplayFake-CN training set) remain harder than synthesis model shifts. This suggests future work should focus on device-agnostic feature learning or adversarial training across playback–recording pairs.

The dataset contribution (ReplayFake-CN) is substantial and addresses a clear gap: most public benchmarks are English-only or lack diverse consumer devices and smart wearables. Release planned on HuggingFace.