COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning

Contributions

• First automated method for detecting out-of-context image use in misinformation
• Self-supervised learning framework: trains on matched/non-matched image-caption pairs without explicit out-of-context annotations
• Large-scale COSMOS dataset of 200K images with 450K captions from news websites, blogs, and social media posts
• Achieves 85% out-of-context detection accuracy; improves over caption-only and image-only baselines

Method

The core approach leverages image-text alignment to detect out-of-context scenarios. At training time, the method pairs each image with its original caption (match) and captions from other images (non-matches), then learns to score how well objects in the image align with the caption text.

Architecture: A Mask R-CNN backbone detects object bounding boxes in the image. Each detected object region is encoded using a pretrained object encoder (ResNet-50 + RoIAlign). The caption is encoded using a pretrained Universal Sentence Encoder. A scoring function computes similarity between each object embedding and caption embedding. The model learns via a max-margin loss to assign higher scores to aligned object-caption pairs.

Text preprocessing: Named entities in captions (person names, locations, organizations) are replaced with hypernyms (e.g., "Robert Grizz Maguire" → "Person", "Granby" → "location"). This allows the model to focus on semantic relationships rather than memorizing named entities.

Test-time detection: Given an image and two candidate captions, the method:

1. Detects objects and computes visual correspondences for both captions
2. Uses a pretrained SBERT model to measure semantic similarity between the two captions
3. Classifies the image as out-of-context if caption alignment differs (one caption aligns with objects, the other does not) or if both captions are semantically different

Results

Quantitative: The model achieves 85% out-of-context detection accuracy. Ablation studies show:

• Text embeddings (especially sentence embeddings from SBERT) matter more than object-level features alone
• Using more training data (160K vs. 16K images) improves accuracy from 72% to 85%
• Outperforms three baselines: EANN (57% accuracy), EmbedNet (59%), and Jin et al. (60%)

Qualitative: Visual groundings demonstrate that the model learns to associate captions with semantically corresponding objects (e.g., linking "buses" in text to vehicles in the image), and correctly identifies mismatches where objects in the image do not correspond to caption claims.

Notes

A solid contribution to the emerging space of visual misinformation detection. The self-supervised learning strategy is practical: the method requires only naturally captioned images (freely available) rather than expensive manual annotations of out-of-context scenarios. The large COSMOS dataset will be useful for benchmarking. One limitation: the model focuses on English-language captions and trained on news/social media images; robustness to other languages and domains unclear.