MMMMM: A Unified Taxonomy for Investigating the Mechanisms of Multilingual MultiModal Misinformation

Authors: Nadav Borenstein, Greta Warren, Desmond Elliott, Isabelle Augenstein

Venue: arXiv cs.CL, 2026 — arxiv:2608.29681

TL;DR

Multimodal misinformation (images paired with misleading text) is poorly understood compared to text-only fake news, hampered by fragmented taxonomies and the complexity of annotating cross-modal relationships at scale. This work proposes a unified taxonomy of multimodal misinformation mechanisms across six orthogonal axes (Classification, Image Type, Rhetorical Role, Topic, Emotion, Multimodal Mechanism) grounded in real-world Twitter/X instances across seven languages, then operationalizes it via a Vision Language Model (VLM) annotation pipeline to enable large-scale analysis of how deceptive mechanisms vary by language and narrative.

Contributions

• Unified taxonomy with 25 distinct mechanisms organized across six axes: distinguishes mismatch and manipulation patterns, image manipulation types (deepfakes, AI-generated, staged), emotional and rhetorical dimensions, and topic-narrative interactions in a single framework.
• High-quality multilingual dataset of 26,979 real-world multimodal misinformation instances from Twitter/X Community Notes and professional fact-checkers, spanning seven languages (English, Spanish, Portuguese, Japanese, French, German, Hebrew).
• Scalable VLM-based annotation pipeline using Qwen 3.5 27B that achieves 79.3% accuracy on mechanism prediction, comparable to expert annotators (82.3% raw agreement), enabling application to emerging issues and underrepresented languages.
• Cross-linguistic and domain insights: Finds that Portuguese-language misinformation overuses Decorative mechanisms; Japanese predominantly employs Slanted mechanisms (scientific errors and conspiracies); AI-generated and forged images are prevalent in technology and science domains; vaccination-related misinformation relies heavily on scientific errors and textual claims rather than image manipulation.

Method

Taxonomy development: The authors iteratively refined six orthogonal axes through qualitative analysis of Community Notes and prior taxonomies (Dufour et al., 2024; Newman & Schwarz, 2024; McCulloh, 2022):

1. Classification — distinguishes misinformation, ads, spam, stolen content, engagement bait (drawing on Volkova et al., 2019).
2. Image Type — categorizes images by visual content (simple photo, AI-generated, screenshot, document, social media screenshot, etc.), expanded from Dufour et al. (2024).
3. Rhetorical Role — functional relationship between text and image (little relation, close relation, beyond text) following Marsh & White (2003).
4. Topic and Message — high-level narrative (politics, economics, health, sports, law, religion, entertainment, culture) and free-text description of the specific claim.
5. Emotion — emotional tone evoked (joy, sadness, anger, fear, pride, ridicule, curiosity, other) via Weikmann & Lecheler (2022).
6. Multimodal Mechanism — how image and text combine to deceive (e.g., Mismatch: identity/place/event, Manipulation: addition/removal/replacement/textual edits, False image: AI-generated/deepfake/staged/imposter/low-end source, Exaggeration: scientific errors/misrepresentation, Textual claims: decorative or denial of authenticity).

Annotation pipeline: A multi-stage VLM pipeline sequentially predicts each axis, with carefully designed prompts (e.g., for emotion prediction, the system inputs username, claim, and image; for mechanism, it uses the (username, claim, image, note) tuple and a three-turn conversation to determine if the image actively participates in deception). Human evaluation on 100 Community Notes samples achieved κ = 0.67 inter-annotator agreement; two expert authors labeled 630 samples in stratified random selection; non-distractor inter-annotator agreement was κ = 0.44 (lower due to ambiguity in fine-grained categories).

Results

Dataset distribution: Community Notes dataset contains 26,979 (username, claim, image, note) tuples; top-level mechanisms are Slanted (7,000 samples, ~26%), Decorative, Mismatch; AI-generated and forged images occur at comparable rates. Most common sub-mechanisms: Manipulated image – removal, Unreliable source, Other mechanism, Deny authenticity.

VLM performance: Pipeline achieves 79.3% accuracy on mechanism prediction across 630 annotated samples, with human–model agreement on non-distractor pairs at 82.3%, suggesting reliability approaching expert annotators. Image Type and Topic reach high agreement (Jaccard ≥ 0.6–0.7); Emotion and Rhetorical Role show lower agreement (0.33–0.44), reflecting ambiguity in subjective annotation.

Cross-linguistic patterns: - Portuguese: Decorative mechanism overrepresented (Fig. 12). - Japanese: Slanted mechanism dominant, driven by scientific errors/conspiracy subcategories (Fig. 13). - Vaccination-related misinformation relies on scientific errors and textual claims rather than image manipulation. - Athletic rivalry misinformation shows high text-based mechanisms (Manipulated image – removal, Textual claim – misattribution). - Technology/science domains: AI-generated and forged images disproportionately prevalent.

Cross-dataset comparison: When applied to AMMeBA (Dufour et al., 2024), mechanism distributions differ substantially from Community Notes — Mismatch is dominant in AMMeBA (driven by Event/Identity mismatches), while Decorative is less common, reflecting different sampling strategies and fact-checker priorities.

Connections

• Multimodal fake news detection — this taxonomy provides the structural framework for understanding how multimodal signals combine in deception, advancing beyond earlier concatenation-based fusion approaches.
• Image manipulation — the taxonomy's explicit categorization of image manipulation types (deepfakes, staged, AI-generated, low-end source) operationalizes detection of visual forgery in context.
• Misinformation mechanisms — directly instantiates a mechanistic view of how false narratives are constructed and propagated through cross-modal coordination.
• Cross-lingual detection and transfer learning — demonstrates that deceptive mechanisms vary substantially by language and cultural context (Portuguese, Japanese, etc.), challenging one-size-fits-all detection approaches and extending cross-lingual research beyond transfer learning to mechanism analysis.
• Taxonomy and typology — exemplifies best practice in crowdsourcing-resistant taxonomy design via iterative grounding in real-world data and VLM operationalization.

Notes

Strengths: This is the most comprehensive unified taxonomy of multimodal misinformation mechanisms to date, grounded in real-world Community Notes data across seven languages and validated via both expert annotation and VLM comparison. The six-axis design elegantly separates concerns (image properties vs. rhetorical function vs. narrative topic) and enables fine-grained analysis of mechanism–topic interactions absent from prior work. The VLM pipeline is pragmatic: it trades marginal expert accuracy for scalability and multilingual coverage, and the paper is honest about where agreement is low (Emotion, Rhetorical Role) and why. The cross-linguistic findings are novel and actionable.

Limitations: (1) Community Notes is biased toward instances that Twitter/X users flagged as misinformation; it misses undetected falsehoods and overrepresents widely-flagged content (sports, politics). (2) The dataset is contemporary and dynamic — patterns may shift as platforms and tactics evolve. (3) Some fine-grained taxonomy axes (Emotion, Rhetorical Role) exhibit lower inter-annotator agreement (0.33–0.44), limiting confidence in those subcategories for downstream task modeling. (4) The paper applies the taxonomy at scale but defers to future work any downstream task (e.g., automated detection or mitigation) that uses the taxonomy as a supervised signal. (5) Several prior taxonomy axes were excluded due to ambiguity or external-data dependence (e.g., Intent from Volkova et al., 2019; Recipient from Wardle & Derakhshan, 2017), limiting comparability with some prior frameworks.

Future work: Extend to video and audio modalities (mentioned as feasible but not implemented); improve fine-grained Emotion and Rhetorical Role agreement via clearer definitions or multi-annotator consensus; apply the taxonomy to mitigation strategy design (which mechanisms are most effectively countered by different interventions); investigate whether VLM predictions on low-agreement axes (Emotion, Rhetorical Role) remain useful for downstream tasks despite subjective variability.