Emergent: a novel data-set for stance classification

William Ferreira; Andreas Vlachos

In brief

The Emergent dataset contains 300 rumored claims and 2,595 associated news article headlines labeled with stance (for, against, observing) and veracity (true, false, unverified), derived from journalists' fact-checking workflow. Using logistic regression with features from headlines and claims, the authors achieve 73% stance classification accuracy — 26% higher than prior textual entailment approaches — establishing a real-world benchmark for fact-checking NLP.

Contributions

Novel dataset of 300 claims and 2,595 article headlines from the Emergent rumor debunking project, labeled by journalists during normal fact-checking workflow
Real-world, naturally occurring data without artificial annotation cost
Article-level stance labels (for/against/observing) distinct from claim-level veracity labels, enabling multiple NLP tasks
Stance classification system using logistic regression with syntactic, alignment, and paraphrase features
Demonstrates that features beyond bag-of-words (dependency parse distance to refuting words, word alignment via PPDB, SVO triple matching, word2vec similarity) substantially improve accuracy

Method

Stance classification is framed as 3-way multiclass logistic regression. Features extracted from article headlines include bag-of-words (BoW), presence of question marks, and RootDist — the minimum dependency parse distance from the sentence root to hedging/refuting words (deny, claim, presumably, etc.). Claim-headline features are computed by word alignment using the Paraphrase Database (PPDB) and the Kuhn-Munkres algorithm, subject-verb-object (SVO) triple matching via PPDB semantic relations (equivalence, forward/backward entailment, independence), and cosine similarity between word2vec representations. L1 regularization enforces feature sparsity, highlighting which features contribute to each stance label.

Results

Test set accuracy reaches 73%, substantially outperforming the lexical overlap baseline (32%) and the Excitement Open Platform textual entailment system (47% when trained on Emergent). Per-class precision/recall: for stance 71%/89%, against 82%/70%, observing 74%/54%. Ablation analysis shows BoW contributes most (5.15% drop), followed by RootDist (2.48%), and PPDB word alignment (0.76%). The observing class is most challenging because observing headlines are syntactically similar to for headlines; both often re-state the claim.

Notes

Emergent is a canonical dataset paper that bridges computational journalism and NLP. Its key strength is derivation from journalists' actual labeling workflow — labels are high-quality and real-world motivated, not crowdsourced. The dataset is balanced across stance classes yet imbalanced toward the for class (47.7%), reflecting how real rumors spread (articles supporting claims outnumber refutations). Claims are source-diverse (Snopes, Twitter, news stories) and the associated articles span multiple outlets with varying stances, making the task realistic. The paper's comparison to textual entailment (EOP) is valuable because stance and entailment are related but not identical — the RTE framing undershoots stance classification, suggesting domain-specific modeling is needed. Future work (mentioned in conclusions) extends to full article stance (not just headlines) and article-level veracity prediction.