NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media

Grace Luo; Trevor Darrell; Anna Rohrbach

In brief

A key misinformation threat is using real images and real captions, but mismatching them to create false narratives — the authors propose the NewsCLIPpings dataset (988 samples) of pristine and automatically falsified image-caption pairs, and benchmark multimodal models (CLIP, VisualBERT) to show that while multimodal reasoning helps, the task remains challenging (66% human accuracy).

Contributions

Introduces the NewsCLIPpings Dataset: large-scale automatically constructed dataset with pristine news image-caption pairs and falsified pairs (real image + wrong caption or vice versa).
Proposes four splitting strategies for generating out-of-context mismatches: (a) caption-image similarity via CLIP, (b) caption-caption similarity via SBERT-WK, (c) person name matching (SBERT-WK), (d) scene matching (ResNet Places365).
Benchmarks state-of-the-art multimodal models (CLIP, VisualBERT) on the dataset across different pretraining domains and backbones.
Provides human evaluation showing pristine samples are trivial for humans (96.2% accuracy) while falsified samples are much harder (35% accuracy), establishing task difficulty.
Applies adversarial filtering to balance dataset difficulty and remove low-quality automatic matches.

Method

The dataset construction proceeds in two stages: (1) Preprocessing — extract image-caption pairs from VisualNews (Liu et al., 2020), label named entities using SpaCy and Rathbucket Entity Linker, compute CLIP embeddings and ResNet scene embeddings. (2) Query-based splitting — retrieve false matches for each pristine pair using four similarity-based strategies: CLIP Text-Image similarity (caption → similar image), CLIP Text-Text similarity (caption → other caption → its image), person-match queries (find images of the same person in different contexts), and scene-match queries (find images of the same scene type). An adversarial CLIP Text-Image filtering step removes implausible falsified samples by ranking matches by similarity and selecting top scorers from a 50-50 split between pristine and falsified distributions, ensuring the task isn't trivial.

For model architectures, CLIP uses separate vision and text encoders with contrastive loss. VisualBERT passes image and text through shared transformer layers with 4M image-text pairs for pretraining. Both are fine-tuned on the NewsCLIPpings splits.

Results

Unimodal model performance: Image-only CLIP achieves 54.71% on Semantics splits; zeroing text shows it's not always necessary, but full multimodal CLIP performs better (66-76% depending on split).

Multimodal model performance (Table 3): On the test set:

CLIP (multimodal): 0.6698 all-samples accuracy, 0.7543 pristine, 0.5853 falsified (Semantics/CLIP Text-Image split)
VisualBERT-CC (pretrained on Conceptual Captions): 0.5413 all, 0.5714 pristine, 0.5307 falsified
VisualBERT-VN (pretrained on VisualNews): 0.5774 all, 0.6333 pristine (10% boost over standard pretraining, indicating domain-specific knowledge helps)

Split difficulty ranking: Semantics splits (caption-image and caption-caption) are easiest (~66-76% CLIP accuracy); Scene/ResNet splits are moderate (~63-64%); Person split is hardest (~60%).

Human evaluation (200-sample subset): Humans average 65.6% accuracy, with pristine at 96.2% and falsified at 35.0%, confirming the task is genuinely difficult even for humans.

ROC curves (Figures 4–5) show CLIP outperforms VisualBERT across thresholds and is strongest on semantic reasoning; VisualBERT-VN shows small advantages on person-recognition tasks due to named-entity understanding.

Notes

This work is notable for focusing on the most subtle form of image repurposing — real images and real captions, just mismatched. The dataset construction is clever in using similarity-based automatic strategies rather than manual annotation, making it large and realistic. The finding that VisualBERT-VN outperforms generic CLIP when pretrained on news-domain data suggests domain-specific models are promising for this task. The human evaluation's 65.6% accuracy floor is a sobering benchmark: even when told to look for mismatches, humans find it hard, especially on falsified samples (35% accuracy). One limitation: all examples are from Western news outlets (Guardian, BBC, USA Today, Washington Post) in English, so generalizability to non-Western and multilingual news is unclear.