Contributions
- Mouth-movement-based detection: Exploits semantically high-level inconsistencies in mouth dynamics as a signal for detecting fake videos, less susceptible than low-level patterns to common perturbations.
- Lipreading pretraining: Leverages pretraining on visual speech recognition (Lipreading in the Wild) to learn rich mouth embeddings without requiring labeled forgery data.
- Strong generalization: Demonstrates superior cross-manipulation generalization compared to existing methods (Face X-ray, Xception, CNN-GRU, Patch-based, FaceSwap, NeuralTextures).
- Robustness evaluation: Comprehensive analysis of robustness to unseen corruptions (compression, noise, blur, pixelation, block-wise distortions) at multiple severity levels; LipForensics maintains high performance where competing methods degrade.
Method
The architecture consists of three stages:
Preprocessing & Feature Extraction: Video frames are aligned to a canonical mouth location, cropped to 25 grayscale frames (96×96 resolution), and passed through a frozen ResNet-18 pretrained on lipreading, which outputs a 512-D embedding per frame. The feature extractor is kept frozen to avoid overfitting to artifacts and promote transferability.
Temporal Modeling: A multi-scale temporal convolutional network (MS-TCN) processes the embeddings to capture short- and long-term temporal regularities in mouth movements at multiple temporal receptive fields. The network is also pretrained on lipreading, inheriting high-level semantic understanding of mouth dynamics.
Classification: A linear classifier on top of the temporal network outputs binary (real/fake) predictions. Training uses standard cross-entropy loss with data augmentation (zooming and horizontal flipping).
Results
Cross-manipulation generalization (Table 1): On FaceForensics++ HQ, LipForensics achieves 99.7% AUC when trained and tested on the same manipulation method. In cross-manipulation scenarios, it reaches 97.1% average AUC, surpassing Face X-ray (91.1%), Xception (97.9% same method but worse cross), CNN-GRU (79.4%), and others.
Cross-dataset generalization (Table 2): On Celeb-DF-v2, DeepFake Detection Challenge (DFDC), and FaceShifter HQ, LipForensics achieves 82.4% AUC when trained on FaceForensics++ — a strong result given the domain shift.
Robustness to unseen perturbations (Table 3, Figure 4): Under progressive corruption severity (saturation, contrast, block-wise distortion, Gaussian noise, blur, pixelation, JPEG compression), LipForensics maintains 92.5% average AUC, substantially outperforming Xception (78.3%), CNN-aug (84.1%), and Patch-based (81.7%). It is particularly robust to compression and pixelation but more vulnerable to block-wise distortions (73.8% vs. 99.7% baseline).
Learning on compressed data: Training on heavily compressed FF++ achieves 99.8% accuracy on FaceShifter and 94.2% on DeepForensics, showing that freezing the feature extractor and relying on high-level temporal cues makes the approach robust to visual quality variations.
Notes
Strengths: The use of lipreading pretraining is elegant and well-motivated—the frozen features capture semantically meaningful mouth representations that generalize across forgery methods. The robustness evaluation is thorough, systematically testing multiple perturbation types and severity levels. The approach requires no audio and scales naturally to video classification. Code is made available on GitHub.
Limitations: Acknowledged by the authors, the method cannot be applied to still images and may fail if the mouth is occluded or left unaltered. Assumes at least some mouth movement is present in the 25-frame clips. Performance declines on limited-mouth-movement videos. Likely vulnerable to deepfakes that explicitly target mouth manipulation (e.g., speech-driven face reenactment).
Follow-ups: Future work could explore multi-modal fusion (audio + visual mouth), longer temporal windows, and adversarial defenses against mouth-targeted generation attacks. Cross-modality generalization to audio-visual synthesis remains open.