Skip to content
AI-Enabled Human Memory Manipulation: Misleading AI-Generated Summaries Distort Human Memory

AI-Enabled Human Memory Manipulation: Misleading AI-Generated Summaries Distort Human Memory

Authors: Mattea Sim, Yael Eiger, Tadayoshi Kohno Venue: arXiv cs.CY, 2026 — arXiv:2609.28820

TL;DR

Large language models frequently generate erroneous summaries of videos, including critical omissions and hallucinated details. In a human-subjects experiment, participants who read AI-generated summaries containing misleading information were significantly less likely to accurately recall the original event—demonstrating that AI misinformation can distort human memory through the same misinformation effect documented in decades of psychology research.

Contributions

  • Systematic quantification of errors in AI-generated video summaries: 100% of summaries from ChatGPT and Gemini contained omissions, with 51.6% omitting critical details.
  • Categorization of error types in LLM output: omissions (critical and non-critical), critical inaccuracies, non-critical inaccuracies, and hallucinated additions.
  • Empirical demonstration that misleading information in AI summaries distorts human memory recognition, with only 48.8% of participants correctly recalling a traffic sign after reading a misleading AI summary vs. 83.6% with accurate information.
  • Evidence that the misinformation effect operates regardless of whether participants believe the summary was AI- or human-generated.

Method

The paper employs two complementary approaches:

Part 1: AI Summary Error Analysis. The authors analyzed summaries generated by ChatGPT-3.5 and Gemini 2.5 Flash-Lite for two 25-second animated videos (a car-pedestrian accident and a traffic sign scenario). They generated five summaries per video and per model, using consistent prompts designed to generate neutral, factual text summaries. Two researchers independently coded each summary for errors across four categories grounded in the human memory literature: omissions, critical inaccuracies, non-critical inaccuracies, and additions.

Part 2: Human-Subjects Memory Experiment. Participants first watched one of the two videos, then read a 21-sentence AI-generated summary that either accurately or misleadingly described the event. Critically, they manipulated whether participants believed the summary was AI-generated or human-written. After a 24–48 hour delay, participants completed a memory recognition task (12-item multiple-choice questions) assessing recall of the original event, including a critical question about a traffic sign.

Results

AI Summary Errors Are Frequent and Irregular. All summaries contained errors: 100% had omissions, 65% had critical inaccuracies, 80% had non-critical inaccuracies, and 70% had additions. On average, each summary contained 7 errors; the maximum observed was 20+ errors. The most common error was omission—51.6% of summaries omitted critical details. For example, 95% of summaries omitted the single most central detail (the colliding car or the critical traffic sign).

Misinformation Effect in AI Summaries Distorts Human Memory. When participants read summaries containing misleading information (e.g., a wrong traffic sign), they showed a strong misinformation effect: 83.6% of participants correctly answered the critical traffic sign question after reading an accurate summary, but only 48.8% answered correctly after reading a misleading summary (χ² = 93.987, p < .001). This effect was equally strong regardless of whether participants believed the summary was AI-generated or human-written (no significant interaction), demonstrating that human memory is equally vulnerable to AI-generated misinformation.

Trust in AI Does Not Moderate Susceptibility to Misinformation. Participants' self-reported trust in AI chatbots did not significantly predict their susceptibility to misinformation. The misinformation effect was strong in magnitude regardless of how much participants trusted AI or how often they used AI.

Connections

Notes

Strengths: The paper makes a novel connection between LLM errors and the misinformation effect, grounding AI safety concerns in well-established cognitive psychology. The human-subjects experiment is carefully designed, with appropriate attention checks and statistical rigor. The error categorization in AI output is systematic and replicable. Critically, the authors show that the source of misinformation (AI vs. human) does not matter—the cognitive effect is equivalent.

Limitations: The study is limited to two short, simplistic videos and two LLM models. Generalization to longer, more complex videos or other domains (news, medical summaries, legal documents) remains open. The experiment uses an idealized "summarization" task; real-world AI deployment in high-stakes contexts often involves more complex information processing. The authors acknowledge that participants in the "human condition" may have suspected the summaries were AI-generated anyway, though they controlled for this belief post-hoc. The high frequency of errors in even the "accurate" summary condition (due to miscoding) suggests the AI literacy manipulation may have been incomplete.

Open questions: How do these effects scale to longer, more realistic documents? Do warnings about potential AI errors mitigate the misinformation effect? Can interactive correction (e.g., side-by-side original-and-summary) help users resist AI-generated misinformation?

Implications: The work argues for human oversight of AI summaries in high-stakes settings and raises questions about how to inoculate users against AI-generated misinformation. The finding that trust in AI does not affect susceptibility suggests that increasing "AI literacy" alone may be insufficient.