Contributions
- Million-scale benchmark: Over 2.6 million images (1.3M real, 1.35M fake) covering 1,000 image classes from ImageNet, significantly larger than prior datasets.
- Diverse generators: Includes images from eight generative models representing both diffusion-based and GAN-based approaches, capturing the diversity of state-of-the-art image synthesis techniques.
- Comprehensive evaluation: Introduces two evaluation tasks—cross-generator image classification and degraded image classification—to assess detector robustness and generalization.
- Detailed analysis: Provides insights into generator correlation, image content generalization, and the impact of image degradation on detection performance.
Method
The dataset construction leverages 1,000 distinct image classes from ImageNet, generating 1,350 images per class—1,300 for training and 50 for testing. Real images are drawn from ImageNet to maintain class balance. Eight generators are employed: Stable Diffusion V1.4 and V1.5 (diffusion-based), ADM and GLIDE (diffusion with text conditioning), Wukong and VQDM (diffusion on multilingual datasets), BigGAN (generative adversarial), and additional commercial models like Midjourney.
For the cross-generator classification task, ResNet-50 detectors are trained on images from one generator and tested on images from all other generators. This evaluates whether detectors trained on a specific generator generalize to unknown generators.
For degraded image classification, images are subjected to realistic degradation: low resolution (112×112 or 64×64), JPEG compression (quality 65 or 30), and Gaussian blurring. This tests detector robustness in real-world scenarios where images may be compressed or downsampled during transmission.
Results
Cross-generator classification: Detectors trained on Stable Diffusion V1.4 achieve 98.5% accuracy when tested on the same generator, but performance drops to 54.9% on Midjourney and varies widely across other generators (Table 2). This dramatic performance degradation demonstrates severe overfitting to specific generator characteristics.
Degraded image classification (Table 5): ResNet-50 models trained on Stable Diffusion V1.4 achieve 98.8% accuracy on uncompressed test images but only 82.2% on Gaussian-blurred images (σ=5), revealing significant vulnerability to image degradation.
Generator analysis: Frequency domain analysis (Figure 2) reveals that GAN-generated images exhibit regular frequency patterns (grid artifacts), while diffusion models show more natural spectral distributions. These artifacts enable discrimination but may not persist through compression or degradation.
Image content generalization: Models trained on a subset of 10 or 50 image classes achieve 73.6% and 81.8% accuracy on the 1,000-class test set, indicating that detectors depend on specific image content characteristics and fail to generalize to unseen object categories.
Notes
This work addresses a critical gap in synthetic media detection research by providing a million-scale benchmark and comprehensive evaluation framework. The finding that detectors fail dramatically across generators—achieving 99% accuracy on one generator but 54% on another—has important implications for real-world deployment, where the attacker's generative model is unknown. The degraded image classification task realistically models transmission scenarios but reveals that current detectors are brittle and require significant robustness improvements.
The frequency-domain analysis provides interpretable insights into why certain generators are easier to detect than others, though these patterns may be exploited by adversarial image modification. Future work should focus on developing detector architectures that generalize across generators and remain robust to common image degradation operations.