Skip to content

The Limitations of Stylometry for Detecting Machine-Generated Fake News

Authors: Tal Schuster, Roei Schuster, Darsh J Shah, Regina Barzilay Venue: Computational Linguistics, 2020 — arXiv:1908.09805

TL;DR

While stylometry effectively detects impersonation and provenance, it fails to distinguish between legitimate and malicious uses of language models for generating false content. LMs produce stylistically consistent text regardless of veracity, and adding even a single auto-generated sentence within human-written text evades stylometric detection. The authors create benchmarks showing that stylometry-based classifiers cannot reliably detect machine-generated misinformation while humans are equally vulnerable to manipulation.

Contributions

  • Demonstrates that stylometric features cannot distinguish LM-generated false information from true information, because LMs maintain stylistic consistency regardless of content veracity.
  • Introduces two adversarial datasets: one for QA-based text extension attacks (auto-completion) and one for modification attacks (adding/removing negations to alter article semantics).
  • Shows that single auto-generated sentences within human text remain undetected by state-of-the-art stylometry classifiers (62% accuracy on short answer scenarios).
  • Provides human evaluation evidence that people are also fooled by machine-generated misinformation, indicating stylometry's fundamental limitation as a defense.
  • Argues that fact-checking and verification approaches are necessary complements to stylometric detection.

Method

The paper evaluates stylometry-based detection in an adversarial setting where:

  1. QA Extension Scenario: An attacker uses Grover (a neural language model) to auto-complete news articles by appending question-answer pairs. A responsible user accepts only correct answers; the attacker accepts false ones. The task is to classify text segments as "real" or "fake" based on stylometric features.

  2. Modification Scenario: An attacker takes human-written New York Times articles and performs semantic modifications by adding or removing negations (e.g., "The right did/not have 'underfunding'" to change factual meaning). The detector must identify whether statements are original or manipulated.

  3. Detector Architecture: Uses a Grover-Mega discriminator (pre-trained, fine-tuned to 10 epochs on training data) evaluated in both zero-shot and adaptive settings. Evaluates on metrics of precision, recall, F1, and accuracy.

  4. Human Evaluation: About 100 examples per scenario tested on human subjects with different access to external knowledge (one group with external sources, one without).

Results

  • QA Extension (Adaptive): Detector achieves 71% accuracy, but on short answers (≤10 words), accuracy drops to 62%. Human with external sources reaches 84% F1 (better than model); human without external sources reaches only 68% F1.
  • Article Modification (Adaptive): Detector completely fails at m=2 modifications (F1=0.74) and barely improves with m=10 (F1=0.63). The added negations change factual meaning in 60% of misclassified cases.
  • Provenance Detection: Stylometry successfully detects fully generated articles (90% accuracy) and article extensions with single generated sentence (95% accuracy), confirming stylometry works for impersonation.
  • Key Finding: Stylometry fails specifically when the malicious intent is to introduce false information, because both true and false LM-generated text maintain the same stylistic profile.

Connections

  • Related to LLM-generated text detection — shows limitations of stylometric approaches versus content-based/fact-checking approaches.
  • Complements Defending Against Neural Fake News — both examine defenses against Grover-generated content but find stylometry insufficient.
  • Extends the Stylometry literature by identifying failure modes in modern LM-based generation scenarios.
  • Cited by LLM-Generated Misinformation work on the vulnerability of stylometry to adversarial generation.
  • Related to Detection Evasion — demonstrates how LMs can evade stylometric detection by maintaining consistency.

Notes

This paper makes a crucial point: stylometry succeeds for authorship attribution and detecting impersonation because deceptive actors must change their writing style. But with LMs, there is no style change needed — the model maintains stylistic consistency regardless of whether the content is true or false. This asymmetry means stylometry cannot be the primary defense against machine-generated misinformation and underscores why fact-checking, external verification, and content-based methods are essential. The paper is methodologically sound with clear benchmarks and honest reporting of limitations (including human-level performance).