Skip to content
CLRM: Cross-Layer Relation Modeling for Audio Deepfake Detection

CLRM: Cross-Layer Relation Modeling for Audio Deepfake Detection

Authors: Yangyang Qu, Siqing Qin, Kong Aik Lee, Massimiliano Todisco Venue: arXiv cs.SI, November 2026 — arXiv:2611.05678

TL;DR

Audio deepfake detection (ADD) distinguishes authentic speech from synthetic or manipulated speech. Existing methods either select informative layers from self-supervised learning (SSL) speech encoders or aggregate features across layers, but this discards useful variation in how different layers represent genuine vs. spoofed speech. CLRM explicitly models cross-layer relations—encoding discrepancies and interactions between selected layer pairs—and uses these relation inputs alongside original detector scores for improved deepfake detection. Evaluated on five external datasets (ASVspoof 2021 LA/DF, ASVspoof 5, In-the-Wild, Fake-or-Real) with three detector architectures, CLRM achieves relative EER reductions of 7.8–16.7%, demonstrating that explicit cross-layer comparisons provide effective detection signals.

Contributions

  • Explicit cross-layer relation representation: Retains layer summaries (mean and standard deviation of hidden states across time) and encodes pairwise discrepancies and interactions using absolute difference, element-wise multiplication, and cosine similarity. Concatenates these with the retained summaries to form a rich \((8d+1)\)-dimensional input for the classifier.

  • Source-guided detector augmentation: Uses speaker-relative statistics (homophonous utterance pairs) to guide layer-pair candidate construction and detection-based configuration selection. Combines candidate ranking with detection error thresholds to identify layer pairs that maximize separation between genuine and spoofed speech.

  • Controlled empirical evaluation: Demonstrates improvements across three XLS-R detector architectures (XLSR-SLS, XLSR-Conformer-TCM, XLSR-AASIST) on five external evaluation datasets, with identical layer summaries and matched auxiliary-head parameters. Controlled ablations isolate the contribution of explicit cross-layer comparisons.

Method

Overview. CLRM augments a frozen XLS-R-based SSL detector by constructing relation inputs from two selected encoder layers and learning compact heads to score those relations.

Layer summaries. For each layer \(l\), CLRM summarizes the hidden states \(H^{(l)}(x) \in \mathbb{R}^{T^{(l)} \times d}\) (across the \(T^{(l)}\) temporal frames) into:

\[z_l(x) = [\mu_l(x); \sigma_l(x)] \in \mathbb{R}^{2d}\]

where \(\mu_l\) and \(\sigma_l\) are standardized (using per-layer statistics from the training split) temporal mean and standard deviation. This yields a compact \(2d\)-dimensional representation per layer.

Relation representation. For a selected layer pair \((p, q)\), CLRM constructs:

\[r_{p,q}(x) = [\bar{z}_p; \bar{z}_q; |\bar{z}_p - \bar{z}_q|; \bar{z}_p \odot \bar{z}_q; \cos(\bar{z}_p, \bar{z}_q)]\]

where \(|\cdot|\) is absolute value, \(\odot\) is element-wise multiplication, and \(\cos\) is cosine similarity. This pairwise comparison captures how the two layers differ in their view of the utterance—fundamental to detecting artifacts that manifest asymmetrically across encoding depths.

Relation scoring. A feed-forward head (linear projection, GELU, dropout, linear output) learns a detection score \(s_{p,q}^{\text{rel}}(x)\) from this relation representation.

Score fusion. The original detector's SSL-based score \(s_l^{\text{base}}(x)\) and the relation score \(s_{p,q}^{\text{rel}}(x)\) are combined via logistic-regression fusion:

\[s^{\text{final}}(x) = a s^{\text{base}}(x) + b s_{p,q}^{\text{rel}}(x) + c\]

where \(a, b, c\) are learned on the official development split.

Layer-pair selection. Geometry-guided layer-pair candidates are constructed by first computing speaker-relative organization metrics (own-speaker rank, max-other margin) on bonafide training utterances. Candidates are ranked by geometry distance and candidate ranking score; detection-based configuration selection then evaluates pairs on the meta-dev split and selects the pair with lowest Macro EER.

Results

Main results. CLRM reduces mean equal error rate (EER) across all three detector architectures on external datasets:

  • XLSR-SLS: 9.7% relative reduction
  • XLSR-Conformer-TCM: 7.8% relative reduction
  • XLSR-AASIST: 16.7% relative reduction

Cross-dataset generalization. Evaluated on five external test datasets (ASVspoof 2021 LA, ASVspoof 2021 DF, ASVspoof 5 Eval, In-the-Wild, Fake-or-Real). Relation consistently outperforms direct concatenation on four of five datasets, with maximum mean EER of 6.462% (Relation) vs. 7.330% (Concat-512) on ASVspoof 5.

Representation ablations. Full relation representation achieves source Meta-test Macro EER of 0.069%, lower than near-parameter-matched alternatives: - Concat-256 (direct concatenation): 0.117% EER - Concat-512: 0.117% EER - Relation-only (dropping layer summaries): 0.148% EER

Robustness. Under stricter source partitions (utterance-disjoint, speaker-disjoint protocols), CLRM improves Meta-test Macro EER from 0.170% to 0.101% (utterance-disjoint) and 0.107% to 0.045% (speaker-disjoint), demonstrating robustness to source variability.

Connections

  • Related to GraphSeal via multi-layer modeling of deepfake artifacts in SSL encoders; where GraphSeal uses spectral similarity, CLRM explicitly compares layer representations.
  • Extends work on multilayer SSL representations for ADD; complements layer-aggregation approaches (XLSR-SLS, Pan et al. 2024) by modeling layer dependencies rather than replacing them.
  • Relevant to Speaker Verification systems that also rely on SSL encoders; speaker-relative geometry concepts apply to both detection and embedding spaces.

Notes

Strengths: Controlled experimental design with matched parameter counts and rigorous separation of configuration selection (Meta-dev) from evaluation (Meta-test). Thorough ablations isolating the contribution of explicit cross-layer comparisons. Consistent improvements across three detector architectures and five external datasets. Practical applicability: existing XLS-R-based detectors can be augmented by training only relation heads and fusion parameters.

Limitations: Evaluation restricted to XLS-R-300M encoder; generalization to WavLM, wav2vec 2.0, or other SSL models unknown. Speaker labels required for geometry-guided candidate construction; unclear whether detection-based selection alone would suffice. Speaker-disjoint protocol shows much smaller EER improvements than source-disjoint, suggesting speaker-specific artifacts dominate; cross-speaker generalization remains a challenge. Limited analysis of which layer pairs provide most useful signals (some pairs contribute little despite being selected).

Open questions: Why does XLSR-AASIST achieve the largest EER reduction (16.7%) while XLSR-SLS achieves only 9.7%? Does relation modeling capture different artifact types than the base detector? How sensitive is layer-pair selection to the choice of Meta-dev partition? Could curriculum-based or multi-scale layer pairs further improve detection?