Skip to content

Automatic Detection of Fake News

Authors: Veronica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, Rada Mihalcea Venue: arXiv cs.CL, 2017 — arxiv/1708.07104

TL;DR

This paper introduces two novel fake news detection datasets covering seven news domains and analyzes linguistic properties that distinguish fake from legitimate content. Using linguistic features including readability metrics, word categories, and punctuation patterns, the authors build classifiers achieving up to 78% accuracy and benchmark human performance, showing that automated systems can match or exceed human ability at detecting certain types of fake news.

Contributions

  • FakeNewsAMT dataset: 240 legitimate news articles across six domains (sports, business, entertainment, politics, technology, education) with crowdsourced fake versions (31,990 words).
  • Celebrity fake news dataset: 100 pairs of legitimate and fake celebrity news articles collected from web sources (39,940 fake words, 70,975 legitimate words).
  • Linguistic feature analysis: Identifies key differences between fake and legitimate news including use of punctuation, emotional language, cognitive processes, and readability.
  • Cross-domain evaluation: Demonstrates that fake news detection is domain-dependent, with some domains (politics, technology, education) generalizing better across domains than others (sports, business, entertainment).

Method

The authors extract five feature families: (1) n-gram bag-of-words features (TF-IDF encoded), (2) punctuation features (11 types from LIWC), (3) psycholinguistic features from the LIWC lexicon (summary, linguistic, and psychological categories), (4) readability metrics (Flesch-Kincaid, Flesch Reading Ease, Gunning Fog, ARI), and (5) syntax features derived from Stanford dependency parses.

They train linear SVM classifiers with five-fold cross-validation on both datasets. Readability features perform best on FakeNewsAMT (78% accuracy), while punctuation features excel on the celebrity dataset (70% accuracy). A combined feature set achieves 74% on FakeNewsAMT and 73% on celebrity news.

Results

Within-domain performance: FakeNewsAMT dataset achieves 78% accuracy with readability features and 74% with all combined features. Celebrity dataset achieves 70% accuracy with punctuation features and 73% with all features.

Human baseline: Two human annotators achieved 71% and 70% accuracy on FakeNewsAMT (κ = 0.38); 80% and 77% accuracy on celebrity news (κ = 0.45). The system matches or exceeds human performance on most domains.

Cross-domain generalization: Significant accuracy drops when training on one dataset and testing on another (56-61% for best feature sets), indicating domain-specific linguistic properties of deception.

Domain-specific analysis: Within FakeNewsAMT, technology (90%), politics (91%), and education (84%) show strong within-domain accuracy with readability features, while sports (51%) and business (53%) are more challenging.

Linguistic insights

Legitimate news in general domains emphasizes cognitive processes (insight, differentiation), function words, and relative language. Fake news uses more social and positive emotional words, expresses greater certainty, focuses on present/future actions, and uses more adverbs and punctuation.

In the celebrity domain, fake content overuses perceptual words (hear, see, feel), positive emotions, and first-person pronouns, while legitimate celebrity news uses more cognitive language and tentative phrasing.

Connections

  • [[2015-rubin-three-types-of-fake-news|Rubin et al.]] — proposes taxonomy of fake news types (satire, hoaxes, fabrications) that frames this work's focus on serious fabrications.
  • Deception Detection — builds on established linguistic markers of deception from psychology and online fraud detection literature.
  • Foundational for later work in multi-domain fake news detection and cross-domain generalization studies.

Notes

Strengths: The paper's main contribution is two high-quality datasets with clear, multi-domain coverage and careful annotation. The linguistic analysis is thorough, revealing interpretable differences between fake and legitimate news. The human baseline comparison provides important context for model performance.

Limitations: (1) Crowdsourced fake news may differ systematically from naturally occurring fake content; (2) cross-domain performance degradation suggests the approach is domain-dependent, limiting practical generalization; (3) the paper does not explore graph-based or network features, relying entirely on linguistic properties; (4) celebrity fake news dataset is smaller and more domain-specific.

Impact: This work established important datasets that became widely used benchmarks in the fake news detection literature and demonstrated that linguistic cues alone can achieve competitive detection accuracy, informing the field's understanding of deceptive language in news.