Skip to content

All the News That's Fit to Fabricate: AI-Generated Text as a Tool of Media Misinformation

Authors: Sarah Kreps, R. Miles McCain, Miles Brundage Venue: Journal of Experimental Political Science, 2022 — DOI

TL;DR

Three experiments investigate whether AI-generated news text can be credible and influence public opinion. The authors find that GPT-2 models (355M to 1.5B parameters) generate text nearly indistinguishable from human-written baseline news, particularly when politically congruent stories are paired with partisan audiences. Larger models produce marginally more credible text, but partisanship is a stronger driver of perceived credibility than model size. Disclaimers fail to consistently reduce belief in AI-generated misinformation, raising concerns about the scalability of synthetic text as a tool for foreign interference campaigns.

Contributions

  • First systematic evidence that AI text generators can produce credible-sounding news articles competitive with human-written baselines
  • Demonstration that partisanship heavily conditions credibility judgments—individuals find politically congruent synthetic stories nearly as credible as real ones
  • Evidence that larger language models marginally improve perceived credibility, but diminishing returns emerge as scale increases
  • Experimental validation that disclaimers are ineffective at reducing belief in AI-generated misinformation
  • Framework for understanding how text-synthesis capabilities scale as a tool for low-resource, high-velocity misinformation campaigns

Method

The authors conducted three experiments on Amazon Mechanical Turk with over 1,500 respondents across treatments. All experiments used GPT-2 at three scales (355M, 774M, and 1.5B parameters), trained on 40 gigabytes of outbound Reddit links with high "karma" scores (a curation signal).

Experiment 1: Credibility upper bounds. Respondents read synthetic news articles on North Korea policy generated by each GPT-2 model size and a human-written New York Times baseline, selected to be credible. A random story rated the credibility on a four-point Likert scale. The experiment used two treatments to isolate model capacity: minimal inputs (one sentence, or even a single word) and domain-specific inputs. Even the lowest-powered 355M model generated output that human raters deemed nearly as credible as the Times baseline.

Experiment 2: Partisanship and policy opinion. The researchers tested how ideological framing and partisan source cues interact with AI credibility. They selected stories from Fox News (Republican-trusted) and The Huffington Post (Democrat-trusted) about immigration and caravan policy. Half the respondents saw AI-generated text aligned with their ideological priors; the other half saw AI-generated text opposing their priors. The key finding: respondents treated AI text as significantly more credible when it aligned with their political views, and partisanship effects persisted even with disclaimers revealing AI generation.

Experiment 3: Scaling and automation. To assess whether larger models could reduce human curation burden, the authors generated 300 outputs from each model size using minimal inputs (first two sentences of the original Times article as seed). They then asked respondents whether each story was believable, accurate, and clear, and aggregated into a credibility score. The distributions across model sizes showed remarkably similar credibility profiles—the 774M and 1.5B models were statistically indistinguishable from the 355M baseline in perceived credibility, despite their far greater parameter counts.

Results

  • Experiment 1: Even the smallest 355M GPT-2 model produced text with credibility ratings (mean = 2.86 on a four-point scale) nearly as high as the New York Times baseline (mean = 3.35). The differences between the 774M and 1.5B models and the Times baseline were not statistically significant (755M: mean = 2.98; 1.5B: mean = 3.19).

  • Experiment 2: Partisanship dominated credibility judgments. Republicans found politically congruent synthetic news from Fox sources significantly more credible (mean = 3.35, 95% CI = 3.17–3.53) than from The Huffington Post (mean = 2.68, 95% CI = 2.58–2.84). Democrats showed less susceptibility to ideological framing but still exhibited partisan preference. Disclaimers revealing AI generation reduced credibility for the politically incongruent story but had minimal effect on congruent stories, suggesting inoculation effects are weaker when prior beliefs are affirmed.

  • Experiment 3: The credibility distribution across all three model sizes were statistically indistinguishable (Kolmogorov-Smirnov test). The authors conducted a Kolmogorov-Smirnov test comparing cumulative distributions: the 355M vs. 1.5B comparison yielded no significant difference (p > 0.05), contrary to the hypothesis that scaling alone improves perceived credibility.

Connections

  • Related to deepfakes and synthetic media research documenting how AI-generated content can deceive human audiences
  • Builds on synthetic media literature showing increasing difficulties in detection as generation quality improves
  • Extends polarization research on political motivated reasoning and partisan asymmetries in information processing
  • Directly relevant to misinformation and disinformation as tools of foreign interference and election integrity threats
  • Contributes to generative AI understanding of how language models scale in credibility and malicious applicability

Notes

Strengths: The paper makes an important early contribution documenting the credibility of synthetic text at scale. The three-pronged experimental design—measuring upper-bound performance, partisanship interactions, and scaling effects—is well-structured. The focus on minimal inputs (single sentences or words) as prompts aligns with how a real adversary would operate at scale, avoiding the assumption that high-quality human-curated seeds are required. The clear policy relevance and implications for foreign interference campaigns add urgency to the findings.

Limitations and open questions: - The experiments use a single policy domain (North Korea and immigration) and a single task structure (credibility ratings on Likert scales). Generalization to other topics, evidence structures, or evaluation approaches remains unclear. - The paper does not measure detection of synthetic text—only credibility judgments. This leaves open how easily readers might identify AI-generated content if alerted to do so. - The disclaimer experiments are limited in scope. More sophisticated interventions (e.g., inoculation priors, improved visual warnings) might prove more effective than the tested approaches. - No measurement of behavioral intent or downstream action (e.g., whether credible-sounding synthetic text actually changes voting, sharing, or political attitudes in realistic social media settings). - The study predates GPT-3 and later larger models; scaling effects may differ with contemporary architectures and training datasets. - No exploration of how source uncertainty (reader ignorance of outlet affiliation) interacts with AI generation.

Significance: This paper presaged the 2023–2024 wave of research on LLMs and misinformation. It provided early experimental evidence that the combination of synthetic text generation and partisan motivated reasoning poses a real threat, shifting the research agenda from pure detection (can we tell AI from human?) to inoculation and resilience (how do people respond to synthetic content and what interventions work?). The finding that disclaimers fail underscores that technological or algorithmic detection alone is insufficient.