Verifying Multimedia Use at MediaEval 2015

Christina Boididou; Katerina Andreadou; Symeon Papadopoulos; Duc-Tien Dang-Nguyen; Giulia Boato; Michael Riegler; Yiannis Kompatsiaris

In brief

This paper introduces the Verifying Multimedia Use task at MediaEval 2015, a benchmark for automatically detecting manipulation and misuse of multimedia content in social media. The task provides a corpus of real-world tweets with images labeled as genuine or misleading, derived from 11 major news events, and defines evaluation methodology using precision, recall, and F-score for binary classification of fake vs. real multimedia.

Contributions

Formal task definition for multimedia verification: distinguishing between manipulated images, reposted content out of context, and legitimate multimedia representations of events
Large-scale verification corpus: 176 real and 185 misused images associated with 12,040 total tweets from 11 real-world news events
Feature extraction pipeline: tweet-based features (count, URLs, hashtags, mentions), user-based features (followers, verification status), and forensic features (JPEG compression, PRNU analysis)
Standardized evaluation framework using IR metrics on balanced development and test sets

Method

The task collects tweets around major news events using keywords and hashtags. Unique images are manually verified as either genuine representations of the event or misuse (digital manipulation or reposting from a different context). Near-duplicate image matching (VLAD-based visual search with high-precision threshold) extends the corpus beyond exact URL matches. Three feature categories are extracted for classification: tweet properties, user profile statistics, and image forensics (double JPEG compression artifacts, PRNU fingerprints).

Results

The development set comprises 9,040 tweets (5,008 real, 7,032 fake) from 11,525 unique users across 11 events. The test set contains 3,781 tweets (1,217 real, 2,564 fake). Events include Hurricane Sandy, Boston Marathon bombing, Sochi Olympics, Malaysia Airlines Flight 370, and several internet hoaxes (Columbian Chemicals, Passport Hoax). Systems are evaluated on binary classification accuracy (precision, recall, F-score) of the fake tweet class. The task allows classification as "unknown" when confidence is low, trading recall for higher precision.

Notes

This is a task overview paper, not a novel methodology contribution. Its value lies in formalizing the multimedia verification task, providing a large manually-verified dataset, and enabling reproducible benchmarking. The corpus became widely used in subsequent multimedia forensics research. The distinction between manipulation (tampering) and misuse (contextual reposting) remains important for real-world verification systems.