Face tampering detection

Face tampering detection focuses on identifying images where a person's face has been manipulated, spliced from another image, or swapped with another identity. This includes both face swapping (replacing one face with another using morphing algorithms) and face splicing (cutting and pasting a face region between images).

Key distinguishing cues

Visual artifacts: Face swapping and splicing operations often leave visual artifacts including: - Stitching artifacts and boundary discontinuities - Unnatural edges (especially around lips and facial hair) - Blurring or smoothing near boundaries to blend the tampered region - Illumination inconsistencies

Low-level signal inconsistencies: - Camera Color Filter Array (CFA) pattern differences between the swapped face and surrounding regions - Local noise residual mismatches due to in-camera processing - Double JPEG compression artifacts (background compressed twice, tampered region once)

Physiological constraints: - Head pose inconsistencies between face landmarks and anatomical impossibilities - Eye gaze direction mismatches - Facial expression incompatibilities with body position

Approaches

Steganalysis-based: Extract low-level features originally designed for steganography detection (e.g., CFA-aware features, noise residuals) and apply SVM or other classifiers. These capture camera characteristics and in-camera processing artifacts.

CNN-based: Fine-tune deep neural networks (e.g., GoogLeNet, ResNet) on tampering datasets to learn high-level visual artifacts. Often combined with preprocessing such as Error Level Analysis.

Multi-stream fusion: Combine high-level visual artifact detection with low-level noise analysis in a single model to improve robustness to post-processing.

Forensic features: Use domain knowledge of image forensics (CFA patterns, JPEG artifacts, illumination models) as hand-crafted or CNN-extracted features.

Key papers

• Two-Stream Neural Networks for Tampered Face Detection — combines visual artifact detection (GoogLeNet) with steganalysis-based triplet network; achieves 0.927 AUC on SwapMe/FaceSwap dataset.
• FaceForensics++: Learning to Detect Manipulated Facial Images — largest facial forgery benchmark with multiple manipulation methods.
• Exposing Deep Fakes Using Inconsistent Head Poses — exploits head pose inconsistencies for deepfake detection.

Datasets

• SwapMe and FaceSwap Dataset — 2010 tampered face images created with SwapMe iOS app and FaceSwap open-source software.

Challenges

• Post-processing robustness: High-quality face tampering includes boundary blurring, blending, and resizing, which remove obvious visual artifacts and challenge forensic techniques.
• Dataset bias: Methods trained on one face-swapping algorithm may overfit to its specific artifacts and fail on others.
• Computational cost: Per-image SVM training or feature extraction can be expensive at scale.
• Small faces: Upsampling artifacts and limited patch coverage make detection difficult for small tampered faces (< 50×50 pixels).

Connections

• Deepfake Detection — broader category including video deepfakes and AI-generated faces
• Image forensics — general image tampering detection techniques
• Steganalysis — low-level feature extraction from images
• Image splicing — detecting composite images