E-FreeM2: Efficient Training-Free Multi-Scale and Cross-Modal News Verification via MLLMs

Authors: Van-Hoang Phan, Long-Khanh Pham, Dang Vu, Anh-Duy Tran, Minh-Son Dao

Venue: SCID '25 (2nd Workshop on Security-Centric Strategies for Combating Information Disorder), August 25–29, 2025, Hanoi — DOI

TL;DR

E-FreeM2 is a training-free, retrieval-based multimodal system for out-of-context (OOC) misinformation detection that leverages pretrained vision-language models (MLLMs) and two-stage chain-of-thought reasoning. By dynamically retrieving cross-modal evidence and ranking via visual similarity, the method achieves 90.0% overall accuracy on the NewsCLIPpings benchmark—outperforming prior SOTA (SNIFFER, 88.4%)—without any trainable parameters, making it robust to adversarial attacks and suitable for edge deployment.

Contributions

• Cross-Modal Data Pipeline: A novel retrieval framework operating on local and global scales with similarity-based filtering (textual + visual) and visual-centric ranking to transform image-caption pairs into high-quality evidence candidates.
• Training-Free Adaptation: E-FreeM2 adapts existing MLLMs (Gemini, GPT-4o mini) for OOC detection via prompt-based two-stage chain-of-thought reasoning without fine-tuning, reducing computational overhead and data-poisoning vulnerabilities.
• SOTA Performance: Achieves 90.0% overall accuracy (90.67% on OOC, 89.49% on NOOC) on NewsCLIPpings—surpassing state-of-the-art methods including SNIFFER—while maintaining zero trainable parameters and 12.77s inference per sample.

Method

Problem Setting: Given an image-caption pair (claim) and external evidence retrieved via search engines, determine whether the image is accurately represented by the caption (in-context) or misrepresented with false narrative (out-of-context).

Stage 1: Multi-Scale Cross-Modal Evidence Retrieval

The system retrieves evidence via two complementary pipelines:

• Textual Retrieval: Extracts captions, article snippets, and metadata from external sources via claim-conditioned search.
• Visual Retrieval: Performs reverse image search to retrieve visually similar images with source metadata and publication timestamps.

Evidence is refined through a Filter Module with two stages:

1. Modality-Specific Similarity Filtering: Computes semantic similarity (all-MiniLM-L6-v2 + cosine) for text and perceptual similarity (ViT embeddings + cosine) for images. Candidates retained if similarity ≥ 0.7.

2. Contextual Relevance Filtering: Applies domain filtering (trusted news outlets only), language filtering (English only), and redundancy reduction (domain-title deduplication).

Visual-Centric Ranking: Final ranking prioritizes visual similarity as the dominant factor (S_final = S_visual), based on the observation that visually similar images better preserve real-world context than potentially manipulative textual reframing.

Stage 2: Two-Stage Multimodal OOC Detection

• Stage 1 (Evidence Validation): Gemini MLLM assesses alignment between the news caption and retrieved candidate evidence, verifying critical elements (entities, temporal/spatial references, narrative coherence).

• Stage 2 (Final Decision): GPT-4o mini integrates Stage 1 verification results with direct scene-level visual analysis. Chain-of-thought techniques include:

• Step-by-step analysis (decomposing task into caption matching, evidence verification, contextual alignment)
• Contextual reasoning (verifying location, time, people, specific events depicted)
• Evidence verification (validating supporting evidence from multiple sources)
• Confidence scoring (0–10 scale, incremental contribution of each evidence piece)

Results

OOC Detection Accuracy (NewsCLIPpings Merged/Balance test set, 7,264 samples):

Method	All	OOC	NOOC
SAFE [Zhou et al. 2020]	52.8%	54.8%	52.0%
EANN [Wang et al. 2018]	58.1%	61.8%	56.2%
VisualBERT	58.6%	38.9%	78.4%
CLIP	66.0%	64.3%	67.7%
DT-Transformer	77.1%	74.8%	75.6%
CCN	84.7%	84.8%	84.5%
SNIFFER	88.4%	86.9%	91.8%
E-FreeM2	90.0%	90.67%	89.49%

E-FreeM2 significantly outperforms SNIFFER by 1.6 percentage points overall and 3.77 points on OOC detection, despite requiring zero trainable parameters (vs. SNIFFER's 99M parameters).

Ablation Studies:

1. Filter Strategies: Combining similarity-based + domain filtering yields highest accuracy across ranking positions (Top-1, Top-2, Top-3), outperforming similarity-only or domain-only approaches.

2. Training Data Utilization: E-FreeM2 achieves 90.0% accuracy with zero training data; SNIFFER requires 100% training data to reach 83% accuracy, demonstrating E-FreeM2's efficiency as a training-free solution.

3. Evidence Component Ablation:

4. Full model: 90.0%
5. Without image evidence: 76.48% (13.52% drop)
6. Without text evidence: 77.05% (12.95% drop)
7. Without domain filters: 56.46% (33.54% drop)

Both image and text evidence are crucial; domain filtering critically prevents low-quality evidence from degrading performance.

Limitations

• Search Engine Dependency: Framework relies on external search engines' availability and quality; evidence retrieval cannot exceed search engine coverage.
• English-Only: Current implementation focuses on English content; multilingual applicability remains limited.
• Limited Evaluation on Other Manipulations: While effective on OOC misinformation, robustness against sophisticated manipulations (deepfakes) has not been extensively evaluated.
• Visual-Centric Bias: While beneficial for OOC detection, visual-centric ranking may underperform in scenarios where textual cues are more critical than visual similarity.
• Latency: Multi-stage filtering and reasoning (12.77s per sample) could be further optimized for real-time applications on mobile/wireless networks.

Connections

• Uses the NewsCLIPpings benchmark dataset for evaluation.
• Shares methodological foundation with multimodal detection via cross-modal similarity and fusion strategies.
• Extends work on fact-checking by automating evidence retrieval and claim validation via MLLMs.
• Addresses out-of-context misinformation as a distinct category from deepfakes and fabricated content.
• Uses vision-language model capabilities similar to LLM-powered fake news detection frameworks.

Notes

Strengths: - Achieves SOTA without fine-tuning; addresses practical deployment constraints on edge devices and reduces vulnerability to training-time adversarial attacks. - Comprehensive two-stage reasoning provides interpretable explanations alongside binary decisions. - Thorough ablation studies justify design choices (visual-centric ranking, dual-modal filtering, domain reliability). - Open-ended retrieval avoids dataset-specific overfitting inherent in fixed-training approaches.

Open Questions: - How does performance generalize beyond English-language content and news sources from Western outlets? - Could adaptive weighting between visual and textual similarity improve performance when textual cues are genuinely more informative? - What is the performance envelope under distribution shift (e.g., when search engines' evidence distributions diverge from fact-checking baselines)?