A Stylometric Inquiry into Hyperpartisan and Fake News

Authors: Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, Benno Stein

Venue: arXiv, February 2017 — arxiv:1702.05638

TL;DR

This paper investigates whether writing style alone can distinguish hyperpartisan news from mainstream and fake news from real news. Using a manually fact-checked corpus of 1,627 articles from BuzzFeed, the authors find that hyperpartisan news is stylistically distinct from mainstream (F1 = 0.78), satire is easily distinguishable from both (F1 = 0.81), but style-based fake news detection alone is insufficient (F1 = 0.46). They introduce a novel application of the Unmasking technique to assess style similarity across text categories.

Contributions

• BuzzFeed-Webis Fake News Corpus 2016: A large manually fact-checked dataset of 1,627 articles (826 mainstream, 256 left-wing hyperpartisan, 545 right-wing hyperpartisan) with detailed fact-checking annotations and publication metadata
• Adaptation of Unmasking for style categories: First application of the Unmasking meta-learning approach to assess style similarity between broader text categories (political orientations) rather than individual authors
• Stylistic analysis of hyperpartisanship: Empirical evidence that left-wing and right-wing hyperpartisan news share significant stylistic similarities despite opposing political views, suggesting a common "style of extremism"
• Satire detection via style: Demonstration that satire can be reliably distinguished from both fake and real news using stylometric features (F1 = 0.81)
• Systematic feature selection framework: Careful selection of stylometric features (n-grams, POS tags, readability scores, dictionary-based features) to avoid overfitting and ensure interpretability

Method

The paper employs stylometric analysis based on writing style features combined with machine learning classifiers (random forests). Their feature model incorporates:

• Lexical features: Character and word n-grams (n ∈ [1,3]), stop words in order
• Syntactic features: Parts-of-speech n-grams (n ∈ [1,3])
• Readability measures: 10 different readability indices (Flesch-Kincaid, Gunning Fog, SMOG, etc.)
• Dictionary features: Word frequency ratios from the General Inquirer dictionaries
• Domain-specific features: Ratios of quoted words, external links, paragraph counts and lengths

Features occurring in fewer than 10% of documents are discarded to avoid overfitting. The Unmasking technique, originally designed for authorship verification, is adapted here to measure style similarity between document sets. In their approach, they iteratively remove the most discriminative style features and plot reconstruction error curves; steeper curves indicate higher style similarity between the two compared text categories.

Results

Hyperpartisanship vs. Mainstream: - Style-based classification achieves 0.75 accuracy with 0.89 recall for hyperpartisan class and 0.78 F1 score - Unmasking curves show that left-wing and right-wing documents have significantly more stylistic similarity to each other than either has to mainstream news - When training on only one side (e.g., left-wing + mainstream), the classifier still generalizes well to the other side (0.74 accuracy on left-wing without right-wing training data), supporting the hypothesis that hyperpartisanship has a shared style

Satire Detection: - Style-based classifier achieves 0.82 accuracy and 0.81 F1 score on the S-n-L News DB (180 articles) - Satire is stylistically distinct from both fake and real news - Outperforms pure topic baselines and is competitive with multi-feature approaches

Fake News Detection: - Style-based detection achieves only 0.46 F1 score, insufficient for practical deployment - When classifiers are oriented-specific (trained separately on left- or right-wing articles), performance remains poor - The authors note that genuine fake news cannot be reliably detected by style alone, but style-based hyperpartisan detection could serve as a pre-filter

Connections

• Related to rumour and veracity detection through shared interest in misinformation classification
• Precursor to Hyperpartisan news detection which leverages this stylometric approach
• Contributes dataset and methodology to fake news corpus development
• Connected to Authorship Analysis through adaptation of Unmasking technique

Notes

Strengths: - Novel corpus with professional fact-checking annotations; carefully balanced across publishers to prevent bias - Creative adaptation of Unmasking from authorship verification to broader style categories - Honest reporting of results—the authors acknowledge style-based fake news detection alone is insufficient - Systematic feature engineering and feature selection that avoids overfitting

Limitations: - Style-based fake news detection has limited practical value given low F1 of 0.46 - Corpus limited to Facebook publishers during the 2016 U.S. election; may not generalize to other platforms or time periods - Binary fake/real distinction (combining "mostly false" + "mixture of true/false") is coarser than ideal - No comparison with contemporary knowledge-based or context-based detection methods

Follow-ups: - Hybrid approaches combining style with fact-checking or social context could improve fake news detection - Investigating what linguistic patterns constitute the "style of extremism" discovered in hyperpartisan news could inform content moderation - Application to other languages and media systems to test generalizability