WACV 2024 — Watch on YouTube
Key themes
The fairness problem in deepfake detection: Existing deepfake detectors show inconsistent performance across demographic groups (gender, age, ethnicity). False positive rates reveal alarming disparities: female Asian and female African individuals are misclassified as fake at rates 1.5–3× higher than male Caucasians, embedding demographic bias at scale in commercial detection systems.
Why standard fairness methods don't directly apply: Deepfake detection differs from typical classification tasks in two key ways: (1) severe class imbalance between real and fake videos, and (2) additional imbalance across demographic groups. Standard fairness methods don't account for both imbalances simultaneously, making them ineffective for this domain.
DFDD (without demographic annotations): When demographic labels are unavailable (the common case), DFDD applies the CVR (CVaR—Conditional Value at Risk) method. The key idea: minimize an empirical version of CVR to ensure all unknown latent demographic groups with at least a specified occurrence probability have low error, even without explicitly knowing group membership.
DFDD+ (with demographic annotations): When demographic labels are available, DFDD+ uses a group-level CVR loss with per-group loss functions designed to address both class imbalances. For each demographic group, compute a CVR-style loss that accounts for real/fake imbalance within that group, then minimize the maximum group-level loss to balance performance across all groups.
Experimental results: Tested on four deepfake video datasets (FF++, Deepfakes DB, DFDC, CelebDF) and five detection models. DFDD and DFDD+ substantially reduce false positive rate gaps between demographic groups (e.g., from 1.93 to 0.96 and 0.65 respectively) while maintaining high detection accuracy and low overall false positive rates.
Practical applicability: The methods are detector-agnostic: they work with any base detection model (ResNet, Inception, etc.) by modifying only the loss function during training. No architectural changes required.
Notes
This work fills an important gap: while deepfake detection has matured technically, the fairness dimension—particularly demographic bias—has been largely neglected until recently. The two-pronged approach (handling both class and demographic imbalance) is methodologically sound, and the empirical results demonstrate substantial fairness improvements. The generality across datasets and detector architectures suggests practical deployability. A limitation acknowledged is the assumption that loss functions can be decomposed; future work should explore how these methods generalize to other forensic domains (audio deepfakes, image manipulation) and whether fairness-aware training improves robustness to adversarial attacks on deepfake detectors.