Contributions
- Incorporates multimodal content (text, image, social context) from microblogs for rumor detection using deep neural networks instead of hand-crafted features.
- Proposes att-RNN, an innovative RNN with a neuron-level attention mechanism that aligns visual features with joint textual and social context representations for effective multimodal fusion.
- Demonstrates significant performance improvements on two real-world datasets (Weibo and Twitter), validating the effectiveness of neural multimodal fusion for rumor detection.
Method
The att-RNN architecture consists of three main components:
Text and Social Context Fusion: An LSTM processes the tweet text word-by-word, with social context features (hashtags, mentions, URLs, sentiment scores) concatenated at each time step. The averaged LSTM outputs produce a joint representation $R_{TS}$ of text and social context.
Visual Feature Extraction: A pre-trained VGG-19 CNN extracts 512-dimensional visual features $R_V$ from tweet images. The visual sub-network uses two fully-connected layers on top of VGG-19 to generate task-specific visual representations.
Attention-based Multimodal Fusion: A neuron-level attention mechanism addresses the challenge that one modality might dominate others. At each time step $m$, the LSTM's hidden state $h_m$ generates an attention vector $A_m \in \mathbb{R}^{512}$ via two fully-connected layers. This attention weights visual neurons: $a_m = \sum_{i=1}^{512} A_m(i) v_i$, producing attention-aggregated visual features $R_V' = [a_1, \ldots, a_n]$. The final multimodal representation $R_I = [R_{TS}; R_V']$ is fed to a binary softmax classifier with cross-entropy loss.
Results
On the Weibo dataset (7,532 training, 1,996 test tweets): att-RNN achieved 78.8% accuracy with F1=0.764 for rumors and F1=0.807 for non-rumors, a 12% improvement over simple feature fusion (66.9%). On Twitter (13,933 training, 991 test tweets): 68.2% accuracy with F1=0.689 for rumors and F1=0.676 for non-rumors, outperforming the visual baseline (59.6%) by 8.6 points.
Component analysis showed visual features contribute a 7% accuracy drop when removed, while social context and attention each account for ~5%. The att-RNN without social context or attention scored 77.2% and 74.5% respectively on Weibo. Comparison with adapted baselines (VQA, NeuralTalk) confirmed att-RNN's superiority: 5% absolute improvements on both datasets.
Notes
Strengths: First systematic study of neural multimodal fusion for rumor detection. Careful attention mechanism for feature alignment. Evaluation on two real-world datasets with objective ground truth (Weibo's official debunking system, Twitter's MediaEval benchmark). Ablation studies clarify each component's contribution.
Limitations: Visual features from generic ImageNet-pretrained VGG-19 rather than task-specific training. The paper assumes images and text relate semantically without explicit hard constraints, relying on implicit learning. Social context features are hand-selected rather than learned. No analysis of failure cases on visually ambiguous rumors.
Follow-ups: Fine-tuning visual networks on rumor-specific data; learning social context features end-to-end; extending to video content; cross-domain transfer learning between Weibo and Twitter.