Contributions
- Reproduces and critically analyzes the experimental setup and evaluation methodology of FNC-1, identifying methodological flaws in the shared task design
- Demonstrates that FNC-1's hierarchical evaluation metric is biased toward majority-class predictions and does not accurately reflect system discrimination ability
- Proposes a new F1-based macro-averaged metric (F1m) that better accounts for class imbalance and yields different system rankings
- Systematically analyzes features used by top-performing FNC-1 systems through ablation studies, identifying which feature groups contribute most to performance
- Proposes a novel architecture: stackLSTM, combining feature engineering with stacked LSTM networks to improve minority-class prediction
- Introduces the Argument Reasoning Comprehension (ARC) dataset as a novel out-of-domain evaluation corpus for stance detection generalization
- Conducts cross-domain experiments showing the importance of proper evaluation for understanding model generalizability
Method
The paper analyzes three top-performing FNC-1 systems (TalosComb, TalosTree, and UCLMR) and proposes improvements through:
Error Analysis: Identifies failure modes including lexical overlap exploitation, complex negation mishandling, and reliance on surface-level cues like definitiveness keywords (e.g., "false," "hoax").
Feature Ablation: Systematically removes feature groups to quantify their contribution. Bag-of-words and character n-grams provide the largest gains; lexicon-based sentiment features surprisingly have minimal impact.
Novel Architecture (stackLSTM): Combines the best feature set from ablation with stacked LSTM networks to capture sequential dependencies. Uses:
- GloVe embeddings for headline and document tokens (max 100 tokens each)
- Two stacked LSTM layers with hidden state size 100 and dropout 0.2
- Concatenation of headline/document features from the last LSTM hidden state
- 3-layer dense network (600 neurons each) with final softmax output layer
Results
On FNC-1 (in-domain): stackLSTM achieves F1m = .802, matching the best system (Athene) and outperforming all other methods. Crucially, it significantly improves performance on the DSC (Discuss) minority class (F1 = .876), which most systems struggle with.
Evaluation metric change: With the proposed F1m metric, system rankings shift substantially. TalosComb and TalosTree, which benefited from FNC-1's biased metric, are outperformed by Athene and stackLSTM.
Cross-domain (ARC→FNC): All models degrade on the out-of-domain Argument Reasoning Comprehension dataset, but TalosComb performs best (F1m = .573 vs. .725 in-domain), suggesting it generalizes better despite overfitting to FNC-1's evaluation scheme.
Human baseline: Five human annotators achieved F1m = .754, establishing an upper bound and confirming the task is genuinely challenging.
Notes
This paper is methodologically significant for the fake-news detection community: it exposes how a widely-used shared task's evaluation design can bias systems toward surface-level heuristics rather than semantic understanding. The finding that lexicon-based features (sentiment, readability) fail to help despite theoretical appeal is particularly noteworthy. The stackLSTM model's superior handling of minority classes via semantic context (sequential LSTM processing) versus lexical features aligns with broader findings that news credibility requires understanding nuance, not just surface cues. The cross-domain experiments introduce a new evaluation paradigm (ARC) and raise important questions about how well FNC-1-trained systems transfer to other stance tasks, foreshadowing the current interest in out-of-distribution evaluation for misinformation detection.