Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions

Contributions

• First large-scale empirical study comparing characteristics of AI-generated versus human-created misinformation through linguistic and stylistic analysis
• Theory-guided approach (Narrative Theory) to systematically generate AI misinformation that captures core narrative elements
• Evaluation showing existing information assessment guidelines have questionable applicability to AI-generated misinformation and that detection models degrade substantially on AI-generated content
• Identification of concerns and implications for practitioners, researchers, journalists, and policy makers regarding AI-enabled misinformation at scale

Method

The study follows a structured pipeline:

1. Dataset compilation: Compiled 12,408 COVID-19 misinformation items from three sources (COVID19-FNIR, CONSTRAINT, COVID-Rumor), covering news stories and social media posts from India, US, and Europe.

2. Extract narrative elements: Applied Narrative Theory to extract core narrative elements (story, discourse elements, evidence) from 250 representative human-created misinformation documents. Two independent coders annotated all 250 documents; inter-rater reliability was high (Cohen's kappa 0.87).

3. Generate AI misinformation: Used GPT-3 with 500 narrative prompts capturing story, purpose, and evidence to generate AI misinformation comparable to human-created content. Collected 175 billion parameters from 45 trillion tokens of human-created text.

4. Analyze linguistic features: Used Linguistic Inquiry and Word Count (LIWC) to examine semantic and linguistic differences. Conducted SAGE (Sparse Additive Generative Model) analysis to identify distinctive expression patterns between AI-generated and human-created misinformation.

5. Evaluate detection models: Tested existing COVID-19 detection models on both human and AI-generated misinformation. Models were trained on CT-BERT, LSTM, GAN, and other architectures.

6. Assess guidelines: Evaluated whether existing information assessment guidelines designed by journalists work on AI-generated misinformation.

Results

RQ1: Characteristics of AI-generated misinformation

AI-generated misinformation exhibited statistically significant differences from human-created content:

• AI-generated news posts were significantly more different from human creations than AI-generated posts (suggesting models produce more variable news)
• AI-generated misinformation showed more emotional and affective expressions, higher cognitive processing keywords, and different communication patterns
• Key linguistic differences: higher use of words related to causation, discrepancy, and authenticity; more analytical thinking and self-focused expressions
• AI-generated posts involved fewer casual expressions and Internet slang, indicated lower authenticity perception
• For affect, AI-generated posts showed more positive and negative emotions with higher salience, especially stronger drive-related expressions

RQ2: Detection model performance

Existing detection models showed significant performance degradation on AI-generated misinformation:

• Models trained on human-created misinformation performed substantially worse when classifying AI-generated content
• This suggests existing labeled datasets and trained models have limited generalizability to AI-generated misinformation
• No single architectural approach (CT-BERT, LSTM, GAN) solved the problem

RQ3: Information assessment guidelines effectiveness

Manual assessment guidelines exhibited questionable applicability:

• Guidelines developed by journalists had limited effectiveness when applied to AI-generated misinformation
• Existing frameworks for identifying misinformation credibility, transparency, and completeness were less effective on AI-generated content
• Assessment showed guidelines were more likely to mimic criteria in AI-generated misinformation than in human-created content

Qualitative themes identified:

1. Enhancement of details (67.8%): AI misinformation specified five Ws (who, what, when, where, why) with vivid stories
2. Communication of uncertainty and limitations (66%): Explicitly acknowledged limitations to increase credibility
3. Tendency to draw conclusions (29.6%): Summarized key points and linked individual cases to generalizations
4. Simulation of personal testimonies (44%): Included human-like emotional appeal and personal experiences

Notes

This is the first large-scale empirical study examining AI-generated misinformation characteristics and existing detection/assessment approaches. The work makes a strong methodological contribution by grounding AI misinformation generation in Narrative Theory, ensuring synthetic content captures genuine narrative structures rather than superficial patterns. The findings are concerning: existing detection models and human assessment guidelines both degrade substantially on AI-generated content, suggesting the field needs new approaches. The COVID-19 context provides a real crisis scenario with high stakes, though the implications extend to any misinformation domain. One limitation is reliance on GPT-3; newer models may produce different patterns. The work opens urgent questions about detection methods and information literacy for an era where LLMs can produce persuasive misinformation at scale.