Skip to content
A Picture Paints a Thousand Lies? The Effects and Mechanisms of Multimodal Disinformation and Rebuttals Disseminated via Social Media

A Picture Paints a Thousand Lies? The Effects and Mechanisms of Multimodal Disinformation and Rebuttals Disseminated via Social Media

Authors: Michael Hameleers, Thomas E. Powell, Toni G.L.A. Van Der Meer, Lieke Bos

Venue: Political Communication, 37(2), 281–301, 2020 — DOI

TL;DR

Multimodal disinformation (paired text and images) is perceived as more credible than text-only misinformation, but fact-checkers can overcome this effect. Importantly, the modality of fact-checks (visual or textual) matters less than their presence; fact-checks are more persuasive for people whose prior attitudes already oppose the false claim, but still provide measurable credibility reductions across partisan divides.

Contributions

  • First experimental study to quantify how visual cues enhance the perceived credibility of disinformation on social media.
  • Demonstrates that multimodal fact-checks are effective rebuttals, despite the enhanced credibility of visual falsehoods.
  • Shows that fact-checker effectiveness is moderated by pre-existing attitudes (motivated reasoning), not just source.
  • Tests effects across two politically-salient topics (school shootings, refugees) with 1,404 U.S. participants.

Method

Design: 2 (Disinformation: text vs. multimodal) × 2 (Source: ordinary citizen vs. news agency) × 3 (Rebuttal: text vs. multimodal vs. none) between-subjects online experiment, conducted August 2018.

Participants: 1,404 U.S. respondents (62.7% completion rate), recruited via SurveySampler; mean age 40.5 years, 61% female, balanced political ideology (44.2% Democrat, 46.2% Republican, 9.7% independent).

Stimuli: Realistic Twitter posts about two topics: - Refugees: Claim that a refugee took a selfie with Angela Merkel and was a "Brussels bomber" terrorist (refuted by PolitiFact). - School shootings: Claim that an armed teacher stopped a school shooting, but the fact-check reveals it was a school resource officer, not a teacher.

Posts varied in source (ordinary citizen vs. CNN or PolitiFact) and rebuttal type. Participants were shown at least 10 seconds per tweet.

Dependent Variable: Perceived credibility of the initial disinformation, measured on five bipolar items (accurate/inaccurate, credible/not credible, true/untrue, reflects reality/distant from reality, contains falsehoods/not—α = .93).

Moderators: Prior attitudes on gun control and refugees; measured separately by topic.

Results

Main Finding 1: Multimodal disinformation is more credible. Visual disinformation on refugees was perceived as significantly more credible (M = 3.48) than text-only disinformation (M = 3.38), t = −2.17, p = .030. Effect was modestly stronger on the emotionally-resonant refugee topic than school shootings.

Main Finding 2: Fact-checks reduce credibility regardless of modality. Exposure to any fact-check (text or visual) significantly reduced perceived credibility of disinformation (M_no-check = 4.29 vs. M_check = 3.47–3.50, p < .001). Multimodal fact-checks were not significantly more effective than text-only fact-checks, contrary to expectations.

Main Finding 3: Motivated reasoning moderates fact-checker effects. Fact-checkers had a stronger effect on perceived credibility among people whose prior attitudes opposed the attitudinal stance of the disinformation (H6). For people already opposed to the false claim, fact-checkers reduced credibility by roughly twice as much as for those holding congruent prior attitudes.

Main Finding 4: Source (news media vs. ordinary citizen) did not affect credibility. Contrary to H2, disinformation from a news source (CNN) was not rated as more credible than disinformation from an ordinary citizen.

Connections

  • Multimodal detection methods focus on computational approaches; this paper provides behavioral evidence that visual elements genuinely increase perceived credibility, motivating technical approaches.
  • Fact-checking effectiveness literature; this study contributes experimental evidence on multimodal rebuttals and motivated reasoning moderators.
  • Correction effectiveness — shows fact-checks work across partisan divides, though effects are stronger for those already opposing the claim.
  • Motivated reasoning — disinformation acceptance is moderated by prior attitudes, with implications for designing interventions.
  • Disinformation spread and social media — Twitter context-specific findings on source credibility and visual persuasion.

Notes

Strengths: - Large, nationally-representative sample with balanced political ideology. - Realistic, contemporary stimuli (actual PolitiFact rebuttals; inspired by real incidents). - Tests two theoretically distinct topics, increasing generalizability. - Rigorous experimental controls (attention check, manipulation checks for visual presence at 98.9% accuracy for refugees, 93.2% for shootings). - Clear policy implications: fact-checkers work, even against visually-framed lies, and strongest when reaching people already skeptical of the false claim.

Limitations: - Twitter-only context; findings may not generalize to other platforms (Instagram, TikTok, WhatsApp) where visual manipulation is more prevalent. - Stimuli were discrete tweets viewed for ~10 seconds in isolation; does not capture organic social media feed exposure or repeated encounters. - Modality of fact-check (visual vs. text) had no meaningful effect, contrary to prediction—suggests that the mere presence of a contradiction may matter more than format. - Motivated reasoning effects are moderate in size; pre-existing attitudes partly limit fact-checker reach, though effects remain significant even for those with congruent prior beliefs. - Study was conducted before widespread deepfakes and video manipulation; visual credibility premium may differ when audiences are aware of fabrication risk.