Skip to content

Neural text generation

Methods and architectures for using neural networks (language models, transformers, RNNs) to generate natural language text, including both beneficial applications (summarization, translation) and potential harms (synthetic fake news, impersonation).

Key observations

Rapid progress in generation quality:
Recent transformer-based models (GPT2, GPT3, BERT-based generators) can produce remarkably coherent and stylistically realistic text, making it increasingly difficult to distinguish machine-generated from human-written content.

Dual-use challenge:
Neural text generation enables valuable applications (machine translation, summarization, conversational AI) but also enables adversarial uses (automated disinformation, impersonation, spam).

Artifacts and biases:
Machine-generated text exhibits characteristic artifacts—exposure bias, variance truncation artifacts, repeated phrases, distributional shift—that can be exploited for detection but also reveal vulnerabilities in model design.

Key papers in this wiki

  • Gehrmann, Strobelt & Rush (2019) — GLTR: Analyzes artifacts in neural text generation through three statistical tests (word probability, token rank, entropy); shows generated text concentrates on high-rank predictions while human text uses tail of distribution; introduces interactive visualization tool that improves human detection from 54% to 72%.
  • Zellers et al. (2019) — Defending Against Neural Fake News: Introduces GROVER, a conditional text generation model for news articles, and investigates both generation and detection of neural fake news; shows exposure bias and variance reduction create detectable artifacts.
  • Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News: Extends neural fake news research to multimodal articles with images and captions. Proposes DIDAN to detect visual-semantic inconsistencies; naive humans detect only 46.2% of generated articles while trained humans reach 67.8% with visual cues.