Skip to content

Generated text detection

The rapid improvement of neural language models (GPT-2, GPT-3, T5, BERT) has created a new detection problem: distinguishing machine-generated text from human writing. This is critical for misinformation and disinformation defense, content moderation, academic integrity, and journalism credibility assessment.

Scope and motivation

Language models trained on large corpora can produce fluent, contextually appropriate text that passes basic human inspection. In abuse scenarios, adversaries can:

  • Generate fake news articles at scale
  • Impersonate real people or sources in comments and social media
  • Create misleading reviews and testimonials
  • Generate convincing propaganda or election manipulation content

Unlike traditional deepfakes (video/audio), text generation requires no specialized equipment and is extremely scalable, making it an asymmetric threat for information ecosystems.

Detection approaches

White-box statistical methods:
When you have access to the generation model's output distribution p(X_i|X_{1:i-1}), you can exploit the fact that models concentrate probability mass on a narrow subset of tokens:

  • Rank-based detection: Generated text uses high-rank (high-probability) tokens much more frequently than human text. GLTR: Statistical Detection and Visualization of Generated Text shows this is the strongest signal: words in the top 100 predictions are 2.41× more common in generated text (under GPT-2) than human text.
  • Probability-based detection: Average per-token log-likelihood or comparison of word probability relative to model's top choice.
  • Entropy-based detection: Entropy of the model's distribution at each position; generated text often has overconfident (low-entropy) predictions followed by top-1 tokens.

These methods are interpretable, require no training, and work across different generation schemes (nucleus sampling, top-k, temperature, beam search).

Black-box discriminative methods:
When you have only the text and a pretrained language model (not the original generator), train a classifier:

  • Text-only features (BERT embeddings, linguistic features)
  • Comparison of likelihood under different surrogate models
  • Fine-tuning BERT/RoBERTa on balanced human/generated corpora

GLTR: Statistical Detection and Visualization of Generated Text demonstrates that white-box rank-based features (AUC 0.85–0.87) outperform traditional bag-of-words classifiers (AUC 0.63).

Key papers

Truthfulness and hallucination detection: - Lin, Hilton & Evans (2021) — TruthfulQA: Benchmark measuring whether language models generate truthful answers; demonstrates larger models are less truthful, generating plausible-sounding falsehoods that mimic human misconceptions. Proposes automated metric (GPT-judge) achieving 90–96% accuracy for predicting human judgments of truthfulness.

Watermarking approaches (proactive detection): - Kirchenbauer et al. (2023) — A Watermark for Large Language Models: Proposes embedding imperceptible watermarks into LLM output during generation. Works by promoting a randomized set of "green" tokens via logit modification; detectable using a z-test without access to model parameters or API. Remains robust under paraphrasing and editing attacks; empirical evaluation shows <1.2% false-positive rate on z≥4 threshold.

Post-hoc statistical detection: - Adelani et al. (2019) — Generating Sentiment-Preserving Fake Online Reviews: Evaluates three automatic detectors (Grover, GLTR, OpenAI GPT-2 detector) on machine-generated fake reviews; demonstrates all detectors achieve high error rates (19.6–40.9% EER) and humans cannot distinguish generated reviews from authentic ones (25–35% accuracy on 4-choice task). - Fagni et al. (2020) — TweepFake: about detecting deepfake tweets: First public dataset of human-written and machine-generated tweets (25,572 tweets; half human, half bot); benchmarks 13 detection methods including BoW, BERT, character encodings, and fine-tuned transformers; finds RoBERTa achieves 90% accuracy but GPT-2 tweets (65-80%) remain difficult to detect; character-level models surprisingly effective for short text. - Dugan et al. (2022) — Real or Fake Text?: Boundary Detection: Reframes fake text detection as identifying where text transitions from human-written to machine-generated (boundary detection) rather than binary classification. Introduces the RoFT game platform and releases 21,000+ annotations across four genres (News, Stories, Recipes, Speeches). Humans achieve 23.4% accuracy on first attempt vs. 10% chance; accuracy improves to 72.3% when allowing top-3 guesses. Key findings: larger models (GPT-2 XL) harder to detect; different genres exhibit different error patterns (common-sense errors in Recipes, generic language in News); monetary incentives improve human learning over time. - Ippolito et al. (2019) — Automatic Detection is Easiest when Humans are Fooled: Empirical study of human vs. automatic detection across three decoding strategies (top-k, nucleus, untruncated random). Fine-tuned BERT achieves 80%+ accuracy on long excerpts versus 71% for expert humans. Critical finding: discriminators trained on one decoding strategy transfer poorly to others (42.5% accuracy drop), whereas humans remain robust. Identifies that detection difficulty is inversely correlated with human-fooling rate. - Solaiman et al. (2019) — OpenAI GPT-2 Release Report: Comprehensive analysis of GPT-2's detection and misuse landscape. Fine-tuned RoBERTa achieves ~95% detection accuracy on 1.5B parameter outputs; human credibility studies show ~75% of large model outputs rated as credible. Examines sampling method effects: nucleus sampling harder to detect than Top-K. Foundational for understanding staged release strategies and detection-generation arms race. - Gehrmann, Strobelt & Rush (2019) — GLTR: Statistical detection and visualization tool showing that language models concentrate on high-rank tokens; human-subjects study demonstrates visual annotation improves fake-text detection from 54% to 72%; widely deployed at gltr.io. - Zellers et al. (2019) — Grover & Defending Against Neural Fake News: Introduces GROVER, a controllable conditional text generator for full news articles; shows humans cannot reliably distinguish GROVER output from hand-written disinformation (2.42/3 vs 2.19/3 trust); GROVER itself detects its own generations with ~92% accuracy, outperforming BERT and GPT-2 baselines.

Broader context

Generated text detection sits at the intersection of multiple research areas:

  • Misinformation & disinformation — if agents can scale fake news generation, detection and prebunking become critical.
  • Synthetic media — parallels detection of deepfake video and audio; shared problem structure of "machine vs. human produced content."
  • Natural language generation — understanding generation artifacts (exposure bias, variance truncation, beam search) informs both generation quality and detection robustness.
  • Content moderation — platforms may need automatic pre-screening for generated content.
  • Academic integrity — essay mills and homework cheating via language models.
  • Journalism & source verification — detecting machine-generated impersonations of real sources.

Open challenges

  • Adversarial robustness: Can generation systems be redesigned to evade detection while remaining fluent? Early results suggest forcing uniform sampling decreases coherence, but this remains an open arms race.
  • Domain adaptation: Methods trained on one domain (news) may not transfer to others (social media, academic writing).
  • Multilingual detection: Most detection methods developed for English.
  • Conditional generation: Generated text that continues from human-written seeds may have different artifacts.
  • Human evaluation at scale: Current evaluations are small (30–50 human subjects). Larger, more diverse populations needed to understand baseline detection ability.