Skip to content
Agentic Mixed-Source Multi-Modal Misinformation Detection with Adaptive Test-Time Scaling

Agentic Mixed-Source Multi-Modal Misinformation Detection with Adaptive Test-Time Scaling

Authors: Wei Jiang, Tong Chen, Wei Yuan, Quoc Viet Hung Nguyen, Hongzhi Yin Venue: ACM Conference, 2026 — arXiv:2603.02519

TL;DR

Vision-language models struggle with mixed-source multi-modal misinformation, where false information combines manipulated images with misleading text or text-image mismatches. This paper proposes AgentM³D, a hierarchical multi-agent framework that decomposes detection into modality-specific agents (textual, visual, cross-modal consistency) coordinated by a planning agent, enhanced with critique-aware best-of-N reasoning and adaptive test-time scaling to achieve zero-shot detection with improved reliability and efficiency.

Contributions

  • Systematic characterization of key challenges in zero-shot multi-modal misinformation detection, including mixed-source misinformation handling and reasoning reliability under uncertainty.
  • AgentM³D: a structured multi-agent framework that hierarchically decomposes M³D into independent modality-specific detection agents with explicit reasoning traces, avoiding error propagation.
  • Adaptive test-time scaling mechanism that uses critique-aware ranking and early-stopping heuristics to balance detection accuracy with computational efficiency, enabling reliable candidate exploration without excessive inference costs.
  • Empirical validation on MMFakeBench and Combined benchmarks (300 Real, 300 TVD, 300 VVD, 300 CMM samples), demonstrating state-of-the-art zero-shot performance across diverse mixed-source misinformation types.

Method

AgentM³D adopts a hierarchical agent architecture designed for mixed-source multi-modal misinformation detection. The framework consists of three main components:

Modality-Specific Detection Agents. A cascade of specialized agents assess textual claims, visual content, and cross-modal consistency independently: - Textual Veracity Detection Agent evaluates whether a text claim contradicts credible evidence, using external knowledge tools for verification. - Visual Veracity Detection Agent assesses whether image content aligns with common sense and verifiable visual facts, using image forensics tools (e.g., forgery detection models). - Cross-modal Consistency Detection Agent verifies semantic alignment between text and image modalities, detecting mismatches or subtle contradictions.

Each agent is tasked as a BernDiT-style zero-shot classification problem, producing both a categorical prediction (real/fake/distorted) and explicit reasoning traces describing its inference process.

Planning Agent and Hierarchical Coordination. A planning agent routes each input through the cascade, determining which reasoning level (single-pass or enhanced best-of-N) to activate based on detected difficulty signals. Modality-specific agents operate independently, progressively refining detection decisions from unmodal veracity assessment to cross-modal consistency checking.

Best-of-N Reasoning with Critique-aware Ranking. For reliable candidate exploration, AgentM³D invokes best-of-N (BoN) reasoning, which samples N independent forward-reasoning trajectories and ranks them using a reward model combined with critique signals. The framework integrates: - A general-purpose reward model that scores reasoning quality based on consistency with ground-truth labels. - Critique-aware scoring: Modality-specific critics (textual, visual, cross-modal) explicitly verify modality-level validity, providing orthogonal signals that improve ranking robustness. - Adaptive early stopping: The Top-m Average Gap metric dynamically determines the minimum number of candidates needed to reach confident scores, reducing unnecessary inference while maintaining accuracy.

The selected candidate's reasoning traces serve as the final agent output, enabling downstream verification and error analysis.

Results

AgentM³D achieves state-of-the-art zero-shot performance on two benchmarks:

MMFakeBench (mixed-source multi-modal misinformation): - Qwen3-VL-4B backbone: 58.0% accuracy (vs. 55.2% for MMD-Agent+BoN baseline) - Qwen3-VL-8B backbone: 62.6% accuracy (vs. 60.1% for MMD-Agent+BoN baseline)

Combined benchmark (unified sampling across Moschek, FakeCellit, VERITE): - Qwen3-VL-4B: 58.0% accuracy - Qwen3-VL-8B: 62.4% accuracy

Ablation studies demonstrate: - Critique-aware ranking increases early-stopping efficiency by ~31% with negligible accuracy loss. - Best-of-N reasoning with N=5 provides balanced accuracy-efficiency trade-off, with BoN-only underperforming when reasoning paths are inconsistent. - Removing the planning agent or cross-modal critique agent significantly degrades accuracy, confirming each component's contribution. - Inference time remains practical: ~69% of samples trigger early stopping on MMFakeBench with mean latency well below unconstrained best-of-5 reasoning.

Connections

Notes

Strengths: - First to systematically characterize zero-shot M³D challenges and provide a structured, modular solution that avoids single-pipeline bottlenecks. - Critique-aware ranking is a novel contribution that decouples reasoning quality from correctness, improving robustness when ground truth is unavailable. - Comprehensive ablations validate each design choice; early-stopping mechanism is practical and efficient without sacrificing accuracy.

Limitations: - Relies on external tools (knowledge bases, image forensics models) whose reliability directly impacts agent decisions; no analysis of tool failure propagation. - Critique agents are lightweight and prompt-based; deeper semantic verification might require fine-tuned critics or richer external signals. - Evaluated only on English-language misinformation; cross-lingual and cross-cultural generalization remain open. - Focus on zero-shot limits applicability to domains where modest in-domain tuning would be preferable.

Follow-ups: - Extending to longer video narratives where temporal consistency becomes a fourth detection dimension. - Incorporating user engagement signals and source reputation into the planning mechanism. - Comparing critique signals from different vision-language models to understand model-specific biases.