Contributions
- First shared task dedicated to hyperpartisan news detection as a binary classification problem
- Two datasets with different annotation strategies: article-level manual annotation and publisher-level distant supervision
- Systematic evaluation of 34 distinct systems on two test splits, revealing that large-scale distant-supervision datasets present greater difficulty than smaller manually-annotated ones
- Analysis of effective features and methods: word embeddings (Word2Vec, GloVe), BERT-based models, sentiment detection, and n-gram features
- Meta-classification experiment showing ensemble methods can outperform single best system
Method
The paper defines hyperpartisan news detection as binary classification: given an article's text and HTML markup, decide if it expresses extreme political bias (either left or right). The task differs from general misinformation detection by focusing on style and framing rather than factual accuracy—hyperpartisan articles mimic regular news form but are one-sided in viewpoint, with opposing views either ignored or attacked.
Two datasets were created: (1) 1,273 articles manually labeled by three annotators on a 5-point Likert scale (then binarized), achieving 0.5 Krippendorff's alpha inter-annotator agreement; (2) 754,000 articles labeled via distant supervision based on publisher bias lists (BuzzFeed News and Media Bias Fact Check), split into public training (600,000) and private evaluation sets. The test set was balanced: 50% non-hyperpartisan, 50% hyperpartisan (half left-wing, half right-wing publishers).
Participating teams employed diverse approaches: hand-crafted n-gram and styloemetric features, word embeddings (Word2Vec, GloVe, ELMo), BERT-based fine-tuning, sentiment and emotion lexicons, named entities, and HTML metadata. Most top teams used neural architectures (CNN, LSTM with attention, hierarchical networks) combined with pre-trained embeddings.
Results
A total of 42 teams participated, producing 34 distinct systems. On the by-article test set, accuracy ranged from 0.462 to 0.822, with precision/recall/F₁ showing diverse trade-offs:
- Best single system (Bertha von Suttner): accuracy 0.822, precision 0.871, recall 0.755, F₁ 0.809 (by-article); accuracy 0.643, F₁ 0.681 (by-publisher)
- Second place (Vernon Fenwick): accuracy 0.820, precision 0.815, recall 0.828, F₁ 0.821
- Third place (Sally Smedley): accuracy 0.809, precision 0.823, recall 0.787, F₁ 0.805
Key findings:
- Word embeddings proved highly effective; standard n-grams also competitive
- BERT-based models showed mixed results (Peter Parker's pre-trained BERT achieved only 0.503 accuracy; fine-tuned BERT from other teams performed better)
- Publication date and hyperlink counts were predictive features
- Ensemble and meta-classification methods outperformed single systems (majority voting: 0.885 accuracy on meta-learning test set)
- By-publisher dataset substantially harder than by-article dataset; top team rank differed between the two splits
Notes
This is a canonical shared-task paper, valuable both as a benchmark for hyperpartisan detection methods and as documentation of the state-of-the-art as of mid-2019. The divergence in performance between by-article and by-publisher test sets is instructive: distant-supervision labels from publisher-level bias lists appear noisier or reflect a different phenomenon than human article-level judgments. The paper's observation that even ensemble methods struggle to exceed ~89% accuracy on the larger dataset suggests genuine difficulty in the task (or mislabeling from the distant-supervision strategy).
One limitation: the paper conflates extreme political bias with hyperpartisanship, and does not deeply address how to distinguish hyperpartisan content from legitimate ideological reporting. The task assumes binary left/right categorization, which may not hold for all political contexts outside the US 2016 election period (the era in which most candidate articles were published).