Disinformation on the Web: Impact, Characteristics, and Detection of Wikipedia Hoaxes

Contributions

• Impact analysis: Quantifies the spread and longevity of Wikipedia hoaxes via pageviews, web citations, and detection time.
• Characterization: Identifies structural and editorial features that distinguish successful hoaxes from legitimate articles, including article length, markup density, link patterns, and creator edit history.
• Automated detection: Develops machine-learned classifiers for hoax detection and survival prediction, achieving 71–98% AUC across multiple classification tasks.
• Human evaluation: Compares human and machine accuracy on hoax detection, showing that humans (66% accuracy) are significantly outperformed by classifiers and exhibit predictable biases.

Method

The authors assembled a labeled dataset of 2,692 Wikipedia hoaxes (articles flagged and deleted by editors) and compared them against ~12,901 non-hoax articles sampled on the same day. They computed three categories of features:

Appearance features: plain-text length, text-to-markup ratio, wiki-link density, web-link density. Successful hoaxes tend to be shorter and contain fewer wiki-links than legitimate articles.

Network features: ego-network clustering coefficient (coherence of the network of linked articles). Hoaxes exhibit lower clustering, suggesting they are less tightly embedded in Wikipedia's editorial network.

Editor and support features: number of prior edits by the article creator, time since creator's first edit, number of prior mentions of the article title in other articles before its creation. Hoaxes tend to be created by newer, less-established editors.

For each task, they trained random forests on balanced datasets and applied forward feature selection to identify the most predictive features. They also ran a human experiment in which MTurk raters compared hoax/non-hoax pairs side-by-side and judged which article was more credible.

Results

Real-world impact: Most hoaxes (88%) survive fewer than one day after creation. However, survival is heavy-tailed: some hoaxes survive for months or longer. Successful hoaxes receive an average of 3.5 pageviews per day and are cited in ~2.1 links from Wikipedia and 1.3 from external websites.

Classification performance: Task 1 (Will a hoax get past patrol?) achieves 66% accuracy and 71% AUC. Task 3 (Is an article a hoax?) achieves 92% accuracy and 98% AUC. Task 2 (How long will a hoax survive?) shows prediction is feasible but noisier, with survival time roughly following a power law. The classifier significantly outperforms humans (86% accuracy vs. 66% for human raters).

Feature importance: Appearance features (plain-text length, text-to-markup ratio) are most predictive across tasks. Editor features (creator's history and edit patterns) also contribute substantially. Network-based features show hoaxes have lower clustering and fewer web-linked references.

Notes

The paper's main strength is its grounding in real-world Wikipedia data and the comprehensive feature analysis across appearance, network, and editorial dimensions. The finding that classifiers significantly outperform humans (86% vs. 66%) highlights the value of automated detection but also raises the concern that sophisticated hoaxes may be harder for both humans and machines to detect. The dataset is limited to English Wikipedia and may not generalize to other languages or platforms. The paper notes that hoax characteristics are somewhat platform-specific: features that matter on Wikipedia (density of wiki-links, editor reputation) may differ from those on news sites or social media. Follow-up work could explore cross-platform hoax characteristics and the semantic content of successful vs. detected hoaxes.