Finding Deceptive Opinion Spam by Any Stretch of the Imagination

Myle Ott; Yejin Choi; Claire Cardie; Jeffrey T. Hancock

In brief

This paper introduces the first large-scale gold-standard dataset of deceptive opinion spam (400 truthful and 400 deceptive hotel reviews) and develops machine learning approaches to detect it. The authors compare three detection strategies—genre identification via part-of-speech features, psycholinguistic deception cues (LIWC), and n-gram text categorization—finding that a combined LIWC+bigrams classifier achieves 89.8% accuracy while humans perform at near-chance levels. The work reveals that deceptive opinions bear linguistic markers related to imaginative writing.

Contributions

First public dataset of gold-standard deceptive opinions: 800 hotel reviews (400 truthful, 400 deceptive), scraped from TripAdvisor and collected via Amazon Mechanical Turk with quality controls.
Comparative evaluation of three detection approaches: genre identification (POS-based), psycholinguistic features (LIWC-based), and text categorization (n-gram models), showing that context-sensitive approaches outperform universal cue detection.
Empirical finding on human performance: human judges perform poorly at deception detection (near random), suggesting deceptive reviewers successfully manipulate readers.
Feature analysis revealing imaginative writing markers: analysis of learned classifier weights shows deceptive opinions correlate with linguistic markers of imaginative rather than informative writing (fewer nouns, more verbs and adverbs, increased first-person singular pronouns).

Method

The authors collected 400 deceptive opinions using Amazon Mechanical Turk, where workers posed as customers for specific Chicago hotels and wrote positive reviews (assuming their boss wanted the hotel marketed favorably). They collected 400 truthful 5-star reviews from TripAdvisor itself for the same 20 popular hotels. This yields an 800-opinion dataset balanced across truthful and deceptive classes.

Three automated detection approaches:

Genre identification: Uses part-of-speech (POS) tag frequencies as features, based on the hypothesis that truthful and deceptive writing occupy different genre spaces (following Rayson et al. 2001).
Psycholinguistic deception detection: Applies the LIWC (Linguistic Inquiry and Word Count) tool, which counts instances of ~4,500 keywords into 80 psychologically meaningful dimensions. Features capture linguistic processes, psychological processes, personal concerns, and spoken categories.
Text categorization: Uses n-gram models (unigrams, bigrams, trigrams) as features, lowercased and unstemmed. The combined LIWC+BIGRAMS model normalizes features before concatenation.

Classifiers: Naïve Bayes and Support Vector Machines (SVM), evaluated via 5-fold cross-validation on 800 reviews.

Results

Automated classifiers significantly outperform humans:

Genre identification baseline: 73.0% accuracy
Psycholinguistic approach (LIWC): 76.8% accuracy
Text categorization (UNIGRAMS+BIGRAMS): 88.4% accuracy
Combined approach (LIWC+BIGRAMS): 89.8% accuracy ← best overall

Human performance (three undergraduate judges and two meta-judges): 56.9%–61.9% accuracy on average, effectively at-chance; humans are biased toward classifying opinions as truthful.

Feature weights reveal systematic patterns: truthful opinions use more nouns, adjectives (comparative), and prepositions, while deceptive opinions use more verbs, adverbs, pronouns (especially singular first-person), and past-participle constructions. Superlative adjectives and singular proper nouns are strong negative signals (more common in truthful reviews).

Notes

This is a landmark dataset paper in opinion spam research. The deceptive opinion spam dataset has been widely adopted as a benchmark for spam and fraud detection work. The psycholinguistic insight—that deceptive reviews exhibit linguistic markers of imaginative rather than truthful writing—has influenced subsequent work in deception detection. A key finding is that standard n-gram features alone outperform psycholinguistic features, though their combination performs best; this suggests that universal deception cues (LIWC) are necessary but insufficient without content-specific signals. Human judges' poor performance is consistent with decades of deception research showing that lay people are poor lie detectors, a critical motivation for automated approaches.