Contributions
- Demonstrates that existing deep learning paradigm for real-vs-fake classification is asymmetrically tuned to detect patterns from training models, failing on unseen generative model families
- Shows that a pre-trained, frozen CLIP-ViT feature space can separate real and fake images well without being explicitly trained for this task
- Proposes two simple baselines (nearest neighbor and linear classification) in the CLIP-ViT feature space that significantly outperform trained deep networks on unseen generative models
- Provides empirical analysis of why prior methods fail: the classifier learns a "sink class" for fake images with low discriminative power across model families
- Demonstrates robustness to post-processing operations (JPEG compression, Gaussian blur) without explicit training
Method
The core insight is that existing fake-image detectors learn asymmetrically: they focus on low-level fingerprints (compression artifacts, color shifts) specific to the training model. When testing on new generative model families, these fingerprints don't exist, and the classifier has no learned features for the real-image distribution.
Feature extraction: The authors use CLIP-ViT-L/14, a large vision-language model pre-trained on image-text pairs. They extract features from the penultimate layer (768 dimensions) without any task-specific training.
Classification methods:
-
Nearest neighbor: Given a test image's feature representation, find the closest real-image and closest fake-image feature in the training set using cosine distance. Predict based on which is closer.
-
Linear probing: Train a lightweight linear layer (+ sigmoid activation) on top of frozen CLIP features using binary cross-entropy loss. This learns only which directions in the feature space separate real from fake.
Both methods are trained on real and fake images from a single generative model (ProGAN) and tested on completely unseen models (LDM, Guided, DALL-E, diffusion variants, autoregressive models).
Baselines compared: ProGAN-trained deep networks, state-of-the-art real-vs-fake classifiers (Blur+JPEG, Inception-based methods), and variants using different backbones (ResNet-50, ImageNet pre-trained).
Results
Performance on same generative model family (ProGAN variants):
- NN + CLIP-ViT: 93-95% accuracy
- Trained deep networks: similar performance
- Both approaches work well when test distribution matches training
Performance on unseen generative model families (LDM, Guided, DALL-E, etc.):
- NN + CLIP-ViT: ~82% average accuracy, +15-20 mAP improvement
- Trained deep networks: drop to 50-55% accuracy (near chance)
- Linear probing + CLIP-ViT: maintains ~82% accuracy, +25.30% improvement over trained baselines
- State-of-the-art baseline (Blur+JPEG): drops to ~85% mAP on unseen diffusion models but only 75.51 mAP overall
Key ablations:
- CLIP-ViT backbone crucial: ResNet-50 and ImageNet pre-trained variants perform significantly worse
- Frozen features superior to training: training the backbone for real-vs-fake task degrades generalization
- Training data source matters: models trained on LAIOM (natural images) generalize better to diffusion models than those trained on ProGAN
- Feature space size: even drastic reduction (720k to 8k images) maintains generalization ability
Robustness to post-processing:
- Linear classifier on CLIP-ViT maintains high AP (>90%) under Gaussian blur and JPEG compression
- Trained deep networks degrade substantially under these operations
- Generalization preserved even with noise
Notes
The work challenges the conventional wisdom that classifiers trained on fake images from one generator can be applied to others. The key insight—that a frozen pre-trained feature space untrained for real-vs-fake discrimination can outperform task-specific training—suggests that domain-invariant properties in large-scale pre-trained models are more useful than task-specific fine-tuning when generalization across diverse generators is needed. The paper is a strong empirical study with clear practical implications for deployment of fake-image detectors in an era of rapidly evolving generative models.