Neural text generation¶
Methods and architectures for using neural networks (language models, transformers, RNNs) to generate natural language text, including both beneficial applications (summarization, translation) and potential harms (synthetic fake news, impersonation).
Key observations¶
Rapid progress in generation quality:
Recent transformer-based models (GPT2, GPT3, BERT-based generators) can produce remarkably coherent and stylistically realistic text, making it increasingly difficult to distinguish machine-generated from human-written content.
Dual-use challenge:
Neural text generation enables valuable applications (machine translation, summarization, conversational AI) but also enables adversarial uses (automated disinformation, impersonation, spam).
Artifacts and biases:
Machine-generated text exhibits characteristic artifacts—exposure bias, variance truncation artifacts, repeated phrases, distributional shift—that can be exploited for detection but also reveal vulnerabilities in model design.
Key papers in this wiki¶
- Zellers et al. (2019) — Defending Against Neural Fake News: Introduces GROVER, a conditional text generation model for news articles, and investigates both generation and detection of neural fake news; shows exposure bias and variance reduction create detectable artifacts.
- Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News: Extends neural fake news research to multimodal articles with images and captions. Proposes DIDAN to detect visual-semantic inconsistencies; naive humans detect only 46.2% of generated articles while trained humans reach 67.8% with visual cues.
Related topics¶
- Generated text detection — detecting machine-generated vs. human-written text
- Fake news detection — detecting machine-generated false content
- Disinformation — understanding and countering false information
- Adversarial Machine Learning — adversarial attacks and defenses in machine learning
- Language Models — large pretrained language models and their properties