Authorship Attribution for Neural Text Generation

Contributions

• Formulates three distinct versions of authorship attribution for neural text generators
• Empirically evaluates eight state-of-the-art NLG models (CTRL, GPT, GPT2, GROVER, XLM, XLNET, PPLM, FAIR) against human-written text
• Demonstrates that linguistic features (word counts, readability metrics, psycholinguistic properties) effectively distinguish human from machine text
• Shows that RoBERTa fine-tuned classifiers outperform simpler models for human vs. machine detection but remain unreliable for certain generators like GROVER
• Identifies that machine generators exhibit distinct stylistic patterns that can be leveraged for authorship attribution

Method

The paper tests three problem formulations:

Problem 1 (Same Method or Not): Given two texts, determine whether they were generated by the same NLG method or one is human and one is machine. The paper creates a balanced dataset where 11% of text pairs share the same generator and 89% differ, tested with RNN and CNN-based encoders.

Problem 2 (Human vs. Machine): Binary classification distinguishing human-written news articles from machine-generated text. Uses news titles from CNN, Washington Post, and NYT as prompts for eight NLG models.

Problem 3 (Authorship Attribution): Multi-class classification identifying which of eight NLG methods (or human) generated a given text.

Datasets and Generators: The authors collected ~1,066 human-written news article titles and generated texts using: CTRL (conditional control codes), GPT (345M parameters), GPT2 (774M), GROVER (trained on political news), XLM (cross-lingual), XLNET, PPLM (plug-and-play language models), and FAIR (Facebook's sequence models).

Feature Analysis: Psycholinguistic analysis using Flesch Reading Ease, Flesch-Kincaid Grade, and LIWC (Linguistic Inquiry and Word Count) with 93 features capturing linguistic dimensions.

Model Architectures: Word embeddings fed into: Embedding + RNN, Stacked CNN, Parallel CNN, CNN-RNN combinations, and classical ML baselines (Naive Bayes, SVM, Random Forest, Decision Trees) with linguistic features and n-grams.

Results

Problem 1 (Same Method): RNN and CNN-RNN achieved F1=0.9813 (balanced setting) and F1=0.7869 (imbalanced), showing that simple models can effectively detect generator similarity even in challenging settings where negatives dominate.

Problem 2 (Human vs. Machine): RoBERTa fine-tuned achieved F1=0.9792 average across individual generators, but performance varied: GPT2 texts were nearly indistinguishable from human (F1=0.6574), while CTRL and XLNET were easiest to identify. GROVER and FAIR posed intermediate difficulty (F1≈0.81). Classical methods combined with linguistic features performed surprisingly well, with Random Forest and SVM achieving F1≈0.94 average across generators.

Problem 3 (Authorship Attribution): Multi-class classification achieved F1=0.5423 macro average using Random Forest, with linguistic features and classical ML outperforming neural methods. Generators like CTRL, GPT, XLM, XLNET were more accurately identified due to distinct stylistic markers, while Human and GROVER texts were consistently confused.

Notes

The paper's main strength is the comprehensive empirical evaluation across eight diverse generators, revealing that while human-vs-machine detection is largely solvable, attribution to specific models remains challenging due to overlapping textual patterns. A notable finding is that GPT2 and GROVER generate text with linguistic properties closer to human writing than other models, suggesting these generators warmed the approach to training data more effectively. The observation that classical ML with linguistic features sometimes outperforms neural classifiers challenges conventional wisdom and suggests that simple, interpretable features remain powerful for this task. Limitations include focus on English political news and relatively small model sizes (2019 era); modern very large language models (GPT-3+) would present a harder problem.