From Skepticism to Acceptance: Simulating the Attitude Dynamics Toward Fake News

Authors: Yuhan Liu, Xiuying Chen, Xiaoqing Zhang, Xing Gao, Ji Zhang, Rui Yan

Venue: arXiv, 2024 — arXiv:2403.09498

TL;DR

This paper introduces FPS, an LLM-based agent simulation framework that models how individuals' opinions on fake news evolve through social interactions. Each agent has personality traits, dual memory (short- and long-term), and reasoning capabilities. Results show political fake news spreads faster than science topics, credulous individuals (high agreeableness, neuroticism) are more susceptible, and early, frequent interventions effectively control propagation.

Contributions

• FPS Framework: First LLM-based simulation framework for fake news propagation that captures semantic information, personality diversity, and human-like reasoning rather than reducing opinions to numerical values.
• Dual Memory System: Agents maintain both short-term memory (daily interactions) and long-term memory (historical context) with reflection mechanisms to mimic human cognition.
• Micro- and Macro-level Analysis: Combines individual opinion evolution tracking with population-level SIS (Susceptible-Infected-Recovered variant) dynamics.
• Intervention Mechanisms: Demonstrates that early and consistently-spaced official refutations (e.g., every 3 days) effectively balance governance cost and effectiveness.
• Empirical Validation: Simulation results align with real-world observations: political misinformation propagates faster than other topics; individuals with specific personality traits are more vulnerable; approximately 50% of infected individuals remain chronically infected despite interventions.

Method

Dynamic Opinion Agent (DOA): Each agent is initialized with a persona (name, age, education level, Big Five personality traits). Agents express opinions as tweet-formatted text rather than numerical scores, enabling rich semantic reasoning. At each timestep, agents engage in random opinion exchanges, reflect on interactions via dual memory, and update their beliefs based on personal traits, memory, and reasoning prompts.

Dual Memory: Short-term memory captures the day's interactions (summarized at day end). Long-term memory integrates today's summary with previous long-term memory, maintaining continuity while managing information volume. This mimics how humans gradually accumulate and refine beliefs.

Reasoning for Opinion Updates: Agents receive prompts like: "You are simulating a real person with [traits] and [education]. Given your [previous opinion] and new information in [long memory], update your opinion. Compose a tweet expressing your belief (0 = disbelief, 1 = belief) with reasoning."

Agent Interaction Simulator (AIS): Manages the social network topology, daily random interactions (c interactions per agent), and intervention mechanisms. Implements a modified SIR model where recovered agents can become infected again, reflecting opinion volatility.

Intervention Mechanism: An official agent issues formal refutations on designated days. Experiments test timing (first day vs. seventh day) and frequency (daily, every 3 days). Early intervention reduces initial spread; frequent intervention maintains control.

Results

Macro-level Observations:

• Topic Comparison (Figure 3): Political fake news reached peak infection (~25 people) by day 4 with no recovery; science topics infected ~10 people with rising recovery rates. Fitted SIS parameters show political topics have transmission rate β ≈ 2× higher and recovery rate γ ≈ 1/10 lower than science.
• Trait Effects: Credulous agents (high agreeableness + neuroticism) showed infection rate 1.733 vs. 0.933 for skeptical agents; recovery rate 0.133 vs. 0.400. Skeptical agents maintained consistent opinions; credulous agents flipped beliefs every 3–14 days.
• Intervention Timing: Early intervention (day 1) reduced belief average from 1.0 to 0.933; mid-stage intervention (day 7) showed slower sustained decline. Early + frequent interventions (daily/every 3 days) achieved belief average of 0.90 with 0.200 recovery rate vs. 1.0 with 0.000 for no intervention.

Micro-level Observations:

Two case studies (Figure 5) illustrate divergent responses to the same fake news: - Michael (credulous, empathic, emotional): Frequently shifted opinions based on others' beliefs ("all believe in his memory"). Reasoning was explicit but swayed by social proof. - Sandra (skeptical, distrustful, placid): Maintained skepticism across 15 days despite encountering differing viewpoints and explicitly storing "differing views" in memory.

Chronic Believers: ~50% of infected agents remained infected throughout, regardless of intervention. Common trait: high agreeableness. Suggests interventions should be tailored to personality profiles.

Connections

• Propagation Models — framework advances agent-based propagation modeling with semantic reasoning instead of numerical dynamics
• Opinion Dynamics — models individual belief evolution via interaction-driven change with personality factors
• [[2023-change-llm-agents]] — related work using LLMs for social simulation
• Misinformation interventions — empirically validates intervention timing and frequency for misinformation control
• Personality Susceptibility — confirms Big Five trait effects on fake news belief (agreeableness, neuroticism)
• Agent-Based Modeling — demonstrates LLM-based agents as viable alternative to traditional SIS/SIR numerical models

Notes

Strengths: - Novel approach using LLM agents to capture semantic and reasoning-level opinion dynamics absent in prior numerical models. - Comprehensive experimental design testing topic, personality, and intervention effects with ablation studies. - Strong alignment with prior empirical findings (political faster than science, trait effects) validates methodology. - Twin macro (SIS fitting) and micro (individual reasoning traces) views provide complementary insights.

Limitations: - Computational cost: each agent's opinion update requires LLM API calls (GPT-3.5-turbo); scalability limited vs. numerical models. Authors report cost in Appendix B/C. - Simplified network: random interaction model doesn't capture real social network structure (homophily, clustering, algorithmic sorting). - Chronic believers (~50% remain infected) suggests model captures resistance but lacks mechanisms to explain why certain individuals are immovable—personality alone may not suffice. - Intervention experiments use synthetic topics (hurricanes, 14th Amendment); generalization to real-world misinformation campaigns unknown. - Lack of validation against real social media cascades; simulation grounded in psychology literature but not directly compared to observed Twitter/Reddit dynamics.

Future Work: - Integrate real social network topologies and compare against observed cascades. - Test on real-world misinformation topics with ground-truth fact-checks. - Extend personality model beyond Big Five; add belief strength, distrust in institutions, etc. - Explore adaptive interventions tailored to agent personality profiles. - Scale beyond ~30 agents per simulation (current bottleneck).