Vision-language models

Vision-language models are neural architectures designed to jointly process and reason over visual (image) and linguistic (text) information. They enable downstream tasks like visual question answering, image-text retrieval, visual commonsense reasoning, and detection of cross-modal mismatches—all relevant to detecting multimodal misinformation.

Key architectures

ViLBERT (Vision & Language BERT): Dual-stream transformer-based model with separate transformer stacks for images and text, connected by co-attentive layers that enable interaction between modalities. Each co-attentive layer computes multi-head attention such that visual tokens attend to textual tokens and vice versa. Pre-trained on Conceptual Captions using masked multimodal modeling and multimodal alignment prediction. Strong zero-shot and few-shot transfer performance across multiple visiolinguistic benchmarks.

CLIP: Contrastive pre-training on image-text pairs from the internet; learns shared embedding space where images and captions with high semantic similarity are close in the embedding space. Enables zero-shot classification by comparing test images to text descriptions of target classes.

BLIP, LLaVA, and other vision-language foundations: Later models that integrate visual encoders with large language models for richer reasoning.

Applications to misinformation

• Satire detection: Li et al. (2020) fine-tuned ViLBERT to detect satirical news by analyzing headline-image pairs; achieved 93.8% accuracy, demonstrating that early fusion and multi-modal pre-training outperform simple feature concatenation.
• Cross-modal mismatch detection: Identifying text-image pairs where the content is incongruent (e.g., a serious political claim paired with an absurd image), a signal of manipulation or satire.
• Fact-checking: Joint visual and textual analysis to verify claims made in news articles or social media posts.

Key papers

• Agentic Mixed-Source Multi-Modal Misinformation Detection with Adaptive Test-Time — AgentM³D: multi-agent framework leveraging vision-language models in a hierarchical cascade for zero-shot mixed-source multimodal misinformation detection, with critique-aware ranking and adaptive test-time scaling.
• Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal — DECEPTIONDECODED: demonstrates that current VLMs are fragile at intent reasoning and overly reliant on superficial alignment cues; introduces intent-guided framework for multimodal misinformation detection.
• Raza et al. (2025) — VLDBench: Evaluating Multimodal Disinformation with Regulatory Alignment — Comprehensive benchmarking of 10 vision-language models on a large-scale multimodal disinformation dataset (31,339 articles, 13 categories). VLMs significantly outperform text-only LLMs (e.g., LLaMA-3.2-11B-Vision: 74.82% vs. LLaMA-3.2-1B: 70.29%). Systematic robustness evaluation reveals VLMs are vulnerable to cross-modal attacks and combined text-image perturbations, highlighting architectural limitations in multimodal consistency detection.
• Ai et al. (2026) — The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detection — Comprehensive survey of LVLMs (GPT-4V, LLaVA, BLIP-2, Qwen-VL, etc.) applied to multimodal fake news detection; proposes three-branch taxonomy and five-dimensional multimodal deception framework; benchmarks across eight datasets with ablations on architectural paradigms, parameter adaptation strategies, and reasoning mechanisms; identifies future research directions on causal reasoning, adversarial robustness, and hallucination mitigation.
• Survey of Hallucination in Natural Language Generation — Survey Section 12 covers object hallucination in image captioning and vision-language model failures in multimodal generation
• A Multi-Modal Method for Satire Detection using Textual and Visual Cues: Fine-tunes ViLBERT for satire detection; demonstrates superiority of early fusion over uni-modal and late-fusion baselines.
• Lu et al. (2019): ViLBERT paper introducing vision-language BERT.
• Radford et al. (2021): CLIP paper (contrastive vision-language pre-training).

Connections

• Multimodal fake news detection — vision-language models are state-of-the-art for multimodal fake news detection.
• Deep learning — transformers, BERT, and pre-training.
• Satire detection — specific application domain.

Notes

Vision-language models benefit from large-scale pre-training on diverse image-text pairs (e.g., Conceptual Captions), which enables transfer to downstream tasks with limited labeled data. However, pre-trained models reflect the biases and knowledge in their training data; they may miss context-dependent satire (e.g., satirical articles that require political knowledge to recognize as fake). Fine-tuning on task-specific data is typically necessary for good performance on misinformation detection tasks.