Contributions
- Presents the first large-scale shared task on stance detection in social media, establishing a standard evaluation benchmark for the community.
- Introduces a dataset of 4,163 tweet-target pairs annotated for stance (favor/against/neither) across five targets for supervised learning (Atheism, Climate Change, Feminist Movement, Hillary Clinton, Legalization of Abortion) and one for weakly supervised learning (Donald Trump).
- Provides formal task formulation distinguishing stance detection from related tasks (sentiment analysis, opinion mining) and clarifying that targets may not be explicitly mentioned in the text.
- Evaluates baseline and 19 participating systems, analyzing the methods, features, and performance across targets (baseline F-score 65.22; best supervised 67.82).
- Demonstrates that stance detection is harder than sentiment analysis and that supervised systems struggle to generalize across targets, motivating future work on transfer learning and domain adaptation.
Method
The task is formulated as a three-way classification problem for each tweet-target pair: the system must determine whether the tweeter is in favor of, against, or neither toward the target. The paper distinguishes stance from sentiment: a tweet may be negative in sentiment but unclear about the target (e.g., a negative tweet about politicians that doesn't specify which politician is the target of opinion).
For Task A (supervised), the dataset comprises 4,163 tweet-target pairs: 70% (2,914 pairs) used for training and 30% (1,249 pairs) for testing. Five targets were used, selected to be well-known in the US and to have a significant volume of favor, against, and neutral tweets. For Task B (weakly supervised), no labeled training data is provided; systems use only 707 test-time tweets and an unlabeled domain corpus of ~78,000 tweets associated with the target "Donald Trump" to infer stance.
Stance annotation was conducted via CrowdFlower with inter-annotator agreement of 73.1%, and a 60% agreement threshold among eight annotators was required for inclusion. The paper observes that stance is often ambiguous when the target is not explicitly mentioned—for example, a tweet about another politician may express opinion relevant to Hillary Clinton without naming her.
Baseline systems include majority class classifier, SVM with word unigrams, SVM with word and character n-grams (1–5 grams), and their combinations. The paper reports that all baselines perform reasonably well (F-score 65–72 for Task A), but most participating systems used additional features such as word embeddings, sentiment lexicons (e.g., NRC Emotion Lexicon), and deep learning models (RNNs, CNNs).
Results
Task A (Supervised): The best result was 67.82 F-score (macro-average of F-scores for favor and against classes) achieved by MITRE, which used two recurrent neural network classifiers trained on task-relevant hashtags from a large unlabeled Twitter corpus. Most participating systems used standard NLP features (n-grams, word embeddings) and some incorporated sentiment features. Results varied substantially by target: systems achieved high scores on Climate Change (72.8 favor, 65 against) and Feminist Movement (64.2 favor, 15.4 against) but lower scores on targets with class imbalance.
Task B (Weakly Supervised): The best result was 56.28 F-score, achieved by pkudblau using rule-based annotation of the domain corpus to train a deep convolutional neural network. Results were substantially lower than Task A (29–56 F-score), reflecting the difficulty of the weakly supervised setting and the imbalanced distribution of favor vs. against tweets in the corpus.
The paper notes that most teams struggled with class imbalance and target-specific generalization, and that per-target metrics reveal that generalizing stance detection across different targets remains an open problem.
Notes
This shared task paper is a canonical reference for stance detection methodology and has been extensively used in follow-up work on misinformation detection, rumor verification, and propaganda identification. The dataset and task formulation have become standard in evaluating stance classifiers. A notable limitation is the small scale (4,163 pairs) and limited diversity of targets, all of which are either political entities or ideological topics, which may limit generalization to other domains. The weakly supervised track (Task B) is particularly relevant for real-world misinformation scenarios where labeled stance data is unavailable. The distinction between stance and sentiment is important and often overlooked in practice.