Human heuristics for AI-generated language are flawed

Maurice Jakesch; Jeffrey T. Hancock; Mor Naaman

doi:10.1073/pnas.2208839120

In brief

In six experiments with 4,600 participants, humans could not distinguish AI-generated from human-written self-presentations beyond chance (50–52% accuracy). A computational analysis revealed that humans rely on flawed heuristics—associating first-person pronouns, contractions, and family topics with human writing—that make their judgments predictable and manipulable. AI systems can exploit these heuristics to generate text perceived as "more human than human," raising urgent questions about scaled automated deception.

Summary figure for Human heuristics for AI-generated language are flawed

Contributions

Demonstrates that human detection of AI-generated self-presentations remains at chance level (50–52%) across professional, dating, and hospitality contexts, unimproved by monetary incentives or feedback training.
Reconstructs the flawed heuristics humans use to detect AI-generated language through qualitative analysis, computational feature extraction, and crowdworker labeling.
Shows that some heuristics (nonsensical content, repetition) are functionally aligned with actual AI-generation patterns, while others (grammatical errors, rare bigrams, long words) are inversely or unrelated to actual source.
Demonstrates that AI systems can be optimized to exploit human heuristics, producing self-presentations rated as human 65.7% of the time—significantly higher than unoptimized AI (51.6%) and actual human text (51.7%).
Proposes "AI accents"—deliberate design choices to make AI-generated language intuitively recognizable to humans—as a mitigating strategy.

Method

Experimental design: Six experiments structured as simplified Turing tests. Participants (N = 4,600 total) read online self-presentation profiles in three contexts—professional (Guru.com job profiles), romantic (OkCupid dating profiles), and hospitality (Airbnb host profiles)—and judged on a five-point Likert scale whether each was human-written or AI-generated. Half of each set was human-authored; half was generated by fine-tuned GPT-2 or GPT-3.

Main experiments: Three contexts, two language models, 7,600 total self-presentations rated (2,000–3,000 per experiment). Hospitality context used fine-tuned GPT-2 (774M parameters); professional and dating contexts used fine-tuned GPT-3 (13B parameters). Experimental treatments included monetary incentives (dating) and trial-by-trial feedback (professional) to test whether increased effort improved detection.

Feature analysis:

Qualitative coding of open-ended participant explanations (two independent coders, 40% cited content, 28% grammar, 24% tone, 8% form).
Computational extraction of ~180 language features: personality measures, sentiment, perspective, readability, emotional language, psychological features (LIWC).
Crowdworker labeling task (N = 1,300): 7,000 self-presentations labeled for nonsensical content, grammatical issues, repetition.
Logistic regression models predicting perceived AI-generation and actual AI-generation from selected features.

Validation experiments: Three additional experiments tested whether AI-optimized self-presentations (selected by trained classifiers to maximize predicted human rating) would be rated as more human than both unoptimized AI text and actual human text.

Results

Detection accuracy: Across three main experiments, participants identified source at 50.3% to 52.2% accuracy—indistinguishable from chance. Neither monetary incentives (51.6%, dating) nor trial-by-trial feedback (51.2%, professional) improved accuracy. No demographic group outperformed others.

Heuristic analysis (Table 1):

Aligned heuristics (participant intuition matched ground truth): nonsensical content (1.10× more likely perceived as AI; 1.23× actually AI); repetitive content (1.08× perceived; 1.47× actual); conversational words (0.95× perceived; 0.90× actual).
Misaligned heuristics (participant intuition inversely predictive): grammatical issues (1.05× perceived as AI but 0.85× actually AI); rare bigrams (1.04× perceived; 0.67× actual); long words (1.03× perceived; 0.78× actual); contractions (0.95× perceived but 1.13× actually AI).
Nonindicative heuristics (used by participants but uncorrelated with source): first-person pronouns, family words, word count, authentic words, past-event focus.

Crowdworker labeling: AI-generated self-presentations rated as nonsensical 13.6% (vs. 9.6% human) and repetitive 12.7% (vs. 7.1% human). AI text had fewer flagged grammatical issues (14.8% vs. 19.6%). A classifier using nonsensical/repetitive labels achieved 58.8% accuracy—outperforming participants by ~7 percentage points—suggesting detectable differences exist but humans do not exploit them effectively.

Prediction models: A logistic regression on 15 selected features predicted participant judgments at 57.6% accuracy on held-out data. A language model classifier (GPT-2 sequence classification head) achieved 58.1% accuracy, showing that flawed heuristics are learnable by machines, making AI-generated text predictable and manipulable.

Validation experiments: Across three contexts, AI-optimized self-presentations were rated as human 65.7% of the time (95% CI), compared to 51.6% for non-optimized AI (p < 0.0001) and 51.7% for actual human text (p < 0.0001). Effect was strongest in professional context (71% for optimized). Optimization used different classifiers per context (regression-based for dating, language-model-based for hospitality, ensemble for professional) to maximize generalizability.

Notes

Strengths:

Large-scale empirical study (4,600 participants, 7,600 judgments) across multiple social contexts, increasing robustness.
Multimethod approach combining qualitative explanations, computational feature analysis, and crowdworker labeling provides independent triangulation of heuristics beyond self-report.
Strong validation design: optimization classifiers trained on main-experiment features successfully produce text perceived as "more human than human," demonstrating predictability and manipulability.
Addresses a timely and consequential problem—the integration of large language models into everyday communication raises real risks of scaled deception.
Proposes concrete mitigation (AI accents) rather than just identifying the vulnerability.

Limitations & Open Questions:

Results limited to current-generation models (GPT-2, GPT-3) and current human heuristics; future model advances or cultural shifts may invalidate findings.
Self-presentations are a narrow, high-stakes genre; generalizability to other AI-generated content (news, code, images) unclear.
Crowdworker labeling task used relatively simple labels (nonsensical, repetitive, grammatical); more fine-grained linguistic features might better explain ground truth.
Validation experiments show exploitability in principle but don't address deployment-scale attacks or sustained deception over time.
No investigation of whether domain expertise (e.g., journalists, hiring managers in the relevant field) improves detection beyond chance.

Significance for the field: The paper challenges the assumption that human-likeness in AI-generated text signals progress in machine intelligence. Instead, it frames inability to detect AI as a sign of human vulnerability to intuitive but misaligned heuristics. This reframes the problem: rather than focusing on making AI harder to detect, the field should consider "AI accents"—deliberate design choices to preserve human intuitive judgment while deploying AI-generated language responsibly.