Contributions
- Largest empirical study to date of conversational AI's persuasive capacity across political beliefs.
- Quantifies persuasiveness of conversational interaction (+2.94pp to +4.1pp over static messages in GPT-4 variants).
- Shows model scale significantly increases persuasiveness; larger models more persuasive than smaller ones.
- Identifies eight rhetorical strategies (moral reframing, storytelling, deep canvassing, information-based argumentation) and measures their persuasive impact.
- Demonstrates information density is the strongest predictor of persuasiveness; more information-dense models make 25+ fact-checkable claims per conversation on average.
- Validates claims against 466,769 LLM assertions using professional fact-checkers; finds persuasive models achieve lower factual accuracy (77% vs. 81% on average).
Method
The study employed a within-subjects design across three experiments with 76,977 UK participants. Participants engaged in 2–10 turn conversations with one of 19 LLMs (ranging from Qwen-1.5 to GPT-4o) about political issues, each prompted to persuade the participant to adopt a pre-specified stance. Conversations were constrained to ~7–9 minutes.
Each experiment examined different factors:
- Study 1: Conversational interaction vs. static AI-generated message (validation of core effect) and model scale effects (17 base models, 4 orders of magnitude in parameter count).
- Study 2: Post-training methods—reward modeling (RM), supervised fine-tuning (SFT), and combinations thereof—to evaluate targeted persuasiveness gains.
- Study 3: Eight rhetorical strategies (moral reframing, storytelling, deep canvassing, information-based argumentation, etc.) and their contribution to persuasiveness.
Persuasiveness was measured as change in agreement from pre-conversation baseline (0–100 scale) on issue-specific statements. Fact-checking used a web-search enabled GPT-4o model validated against 51 professional fact-checkers (r = 0.87, 95% CI [0.84, 0.90]).
Results
Conversational persuasiveness exceeds static messaging: Conversation with AI was substantially more persuasive than exposure to static AI-generated messages, with GPT-4o showing +2.94pp mean persuasive effect vs. GPT-4.5 (+3.08pp) and others ranging +0.1pp to +4.1pp depending on model variant (p < .001).
Model scale drives persuasiveness: Across 17 base models, larger models systematically more persuasive; the association shows +1.59pp Bayesian 95% CI [1.07, 2.13] increase in persuasion per order of magnitude increase in model scale.
Post-training methods increase persuasiveness:
- Reward modeling (RM) added +2.32pp (p < .001) to Llama 3.1-8B, most effective method.
- Supervised fine-tuning (SFT) added +0.26pp (p = 0.230), not significant.
- Combined RM + SFT showed interaction (p = 0.558).
Eight rhetorical strategies ranked by persuasiveness (relative to basic prompt): Information-based argumentation yielded largest persuasive advantage (+2.29pp), followed by Debate (+1.37pp); Moral Reframing, Storytelling, Deep Canvassing, and Norms all near zero or negative. Information prompt ranked highest across studies.
Information density is key: Information density (number of fact-checkable claims per conversation) strongly associated with persuasiveness (Bayesian 95% CI [0.05, 0.99], r = 0.76). Each additional information-dense claim increased persuasion estimate by +0.30pp on average.
Accuracy vs. persuasiveness tradeoff: Persuasive models (especially information-prompted) showed lower factual accuracy; average accuracy across models and conditions was 77/100 (0 = completely inaccurate, 100 = completely accurate). Information-prompted models achieved lower accuracy on average; 29.7% of claims in maximal-persuasion conditions were inaccurate (≤50/100 scale) vs. 16.0% on average.
Notes
Strengths: Exceptionally large sample (76,977 participants), wide model diversity (19 LLMs), rigorous fact-checking validation (professional fact-checkers, 466,769 assertions), and clear experimental design isolating multiple factors (scale, post-training, rhetoric, strategy). The finding that persuasiveness correlates with lower accuracy is critical for policy: AI systems optimized to persuade are not incentivized to be truthful.
Limitations & open questions:
- UK-only sample; cross-cultural generalization unknown.
- Participants knew they were in a study and conversing with AI; real-world persuasion contexts (e.g., chatbots disguised as humans) may differ.
- Focus on text-based LLMs; multimodal AI persuasion not examined.
- Causality of information density remains partly unclear: is information density the primary driver, or is it correlated with other persuasion-relevant features (fluency, coherence)?
- Real-world deployment risk: the most persuasive prompting strategies (information density, certain rhetorical strategies) are also the ones producing the least accurate output.
Implications:
- Red flag for AI safety: conversational AI can reliably persuade political beliefs; persuasiveness and accuracy are decoupled and sometimes negatively correlated.
- Policy relevance: deploying persuasive AI systems without safeguards against misinformation could exacerbate polarization and erode trust in information ecosystems.
- Raises urgent questions about how platforms and developers will balance persuasive capability, factual accuracy, and democratic legitimacy.