Contributions
- Presents a simple, interpretable stance detection system that achieves third place in FNC-1 despite using elementary features (TF/TF-IDF vectors) and a shallow neural architecture (single-hidden-layer MLP)
- Demonstrates that simple systems can be competitive with complex ensemble methods, challenging the assumption that better performance requires sophistication
- Provides open-source implementation on GitHub for reproducibility, establishing a reusable baseline for the community
- Analyzes performance across the four stance labels, identifying that the system excels at related/unrelated classification (96.55% accuracy) but struggles with fine-grained distinctions (agree/disagree)
- Proposes the system as the "simple but tough-to-beat baseline" for FNC-1, encouraging researchers to use it as a comparison point for future methods
Method
The system uses a straightforward pipeline:
Feature extraction: Two parallel bag-of-words representations extracted from headline-body pairs:
- TF vectors: Top 5,000 most frequent words from training set, with English stop words removed (negation terms retained)
- TF-IDF vectors: Top 5,000 most frequent words defined on both training and test sets
- Cosine similarity metric: ℓ₂-normalized TF-IDF cosine similarity between headline and body vectors
These three components (headline TF, body TF, TF-IDF similarity) are concatenated into a feature vector of size 10,001.
Classification: Multi-layer perceptron with one hidden layer:
- Hidden layer: 100 units with ReLU activation
- Output: Softmax layer over four classes (agree, disagree, discuss, unrelated)
- Training: Minimize cross-entropy loss with ℓ₂ regularization (weight decay) and dropout on both layers
- Optimization: Adam optimizer with gradient clipping by global norm ratio
- Implementation: TensorFlow
Hyperparameter tuning: Random search over grid of 20–30 parameter combinations, cross-validated on 50 random train-test splits of the official baseline setup (Table 1 in paper): vocabulary size (5,000), hidden layer size (100), dropout (0.6), regularization strength (0.0001), learning rate (0.01), and gradient clipping ratio (5).
Results
FNC-1 Stage 1 test set (25,413 instances):
- Overall accuracy: 88.46%
- FNC-1 score: 81.72% (official metric)
- Confusion matrix analysis:
- Unrelated class: 97.90% accuracy (18,349 instances) — system nearly perfect at identifying unrelated pairs
- Discuss class: 81.38% accuracy (4,464 instances) — reasonable but problematic
- Agree class: 44.04% accuracy (1,903 instances) — poor performance
- Disagree class: 6.60% accuracy (697 instances) — near-random
Competition ranking: 3rd place out of 50 teams
- 1st place (SOLAT in the SWEN, Talos Intelligence): 82.02%
- 2nd place (Athene, TU Darmstadt): 81.97%
- 3rd place (UCLMR): 81.72%
- Official baseline: 75.20%
The top two teams used ensemble approaches with deep neural networks (convolutional and recurrent) and multiple feature sets, yet achieved less than a 0.3 percentage point improvement over this single, simple system.
Notes
This paper exemplifies an important principle in machine learning systems: simplicity and interpretability can be competitive with complexity when features are well-chosen. The paper's honest acknowledgment of its weaknesses (particularly poor agree/disagree classification) is refreshing and empirically grounded. The system's strength at the unrelated/related distinction suggests lexical similarity features alone are sufficient for binary topic relevance but insufficient for nuanced stance understanding. The authors' promotion of their system as a baseline has likely influenced the field—their philosophy that baselines should be strong enough to motivate future work beyond mere complexity for complexity's sake remains sound. The open-source release accelerated adoption as a comparison point. A limitation is that the paper presents results on only one dataset; cross-domain analysis came later in subsequent work like Hanselowski et al. (2018).