TwiBot-22: Towards Graph-Based Twitter Bot Detection

Contributions

• Largest-scale graph-based Twitter bot detection benchmark: TwiBot-22 contains 1 million users, 860,057 human users, 139,943 bots, and a heterogeneous graph of 92.9M nodes and 170.2M edges—approximately 5× larger than TwiBot-20.

• Heterogeneous entity and relation structure: First bot detection dataset providing 4 types of entities (hashtags, tweets, users, URLs) and 14 types of relations (follower, following, post, mention, retweet, etc.), substantially enriching the network beyond user-only graphs.

• Significantly improved annotation quality: Uses weak supervision learning with expert annotations and Snorkel majority voting, achieving 90.5% accuracy (vs. 80% for TwiBot-20), with expert inter-rater agreement and uncertainty modeling via ensemble learning.

• Comprehensive baseline re-implementation: Re-implements and evaluates 35 representative Twitter bot detection methods (feature-based, text-based, and graph-based) across 9 datasets, providing standardized evaluation and fair comparison framework.

• Unified evaluation framework: Consolidates datasets, preprocessing code, and implemented baselines into the TwiBot-22 framework, enabling reproducible research and lower barriers to entry for future bot detection work.

Method

Data Collection (Diversity-Aware Breadth-First Search):

• Starts with 40 seed users from four domains (politics, business, entertainment, sports)
• Uses distribution diversity and value diversity sampling to ensure neighbors with heterogeneous metadata values are selected
• Expands to approximately 1 million users and their heterogeneous network interactions

Heterogeneous Network Construction:

• Collects four entity types: users, tweets, hashtags, URLs
• Captures 14 relation types including follows, posts, retweets, mentions, contains-hashtag, and contains-URL
• Represents real-world Twitter heterogeneity absent in prior binary user-only graphs

Data Annotation (Weak Supervision Learning):

• Expert annotation: 1,000 randomly selected users assigned to 5 experts with 8:2 training/test split
• Noisy label generation: 8 hand-crafted labeling functions and 7 feature-based/neural-network-based bot detection models
• Label aggregation: Snorkel framework with majority voting (90.5% accuracy on expert set, vs. 80% for TwiBot-20)
• Uncertainty modeling: Removes top 40% uncertain predictions to alleviate label noise

Results

Dataset Scale and Structure:

• 92,932,326 nodes and 170,185,937 edges (largest graph-based Twitter bot detection dataset)
• 4 entity types and 14 relation types (vs. TwiBot-20's 2 entity types and 3 relation types)
• 860,057 human users (60.8%), 139,943 bots (9.9%), 1 million users total
• Significantly improved annotation quality: 90.5% expert agreement vs. 80% for TwiBot-20

Benchmark Findings:

• Graph-based superiority: Top 5 graph-based methods (BotRGCN, GAT, RoBERTa, RGT) outperform all baselines by 13.8% on TwiBot-20 and 8.2% on TwiBot-22, indicating graph structure is critical for bot detection.
• Scalability challenges: Many existing methods fail to scale to TwiBot-22 (e.g., Dehghan et al. achieves near-sota on TwiBot-20 but fails on TwiBot-22), revealing the dataset addresses real-world scalability issues.
• Performance drop on larger dataset: Average accuracy across baselines drops 2.7% from TwiBot-20 to TwiBot-22, suggesting TwiBot-22 captures more challenging, realistic bot behaviors.
• Graph component importance: Removing graph components from graph-based methods substantially degrades performance (e.g., BotRGCN drops from 66.9% to 57.5% accuracy), confirming graph structure's critical role.
• Generalization capability: Graph-based approaches generalize better across train/test splits: BotRGCN achieves best average score (76.50%) in generalization study, suggesting network structure provides robust generalization signal.

Notes

Strengths:

• Unprecedented scale and heterogeneous network structure captures real-world Twitter complexity beyond prior benchmarks.
• Rigorous annotation via experts and weak supervision with Snorkel provides higher-quality ground truth than crowdsourcing alone.
• Comprehensive re-implementation of 35 baselines across 9 datasets provides robust understanding of research progress and fair performance comparison.
• Unified framework (code, datasets, evaluation scripts) substantially lowers barriers for future research.
• Clear finding that graph-based approaches substantially outperform alternatives guides future research direction.

Limitations:

• Annotation from a snapshot in time (2020); bot tactics continue to evolve, potentially limiting long-term generalization.
• Weak supervision labels introduce some noise despite majority voting; false positives/negatives may impact downstream classifier training.
• Generalization study on 10 sub-communities provides evidence but limited exploration of temporal generalization (e.g., do models trained on 2020 data generalize to 2022 bots?).
• Heterogeneous graph structure is richer than TwiBot-20 but still simplified relative to full Twitter feature set.

Open questions:

• How do detection methods perform on more recent Twitter data (2023–2024) when bot tactics have likely further evolved?
• Can the framework incorporate additional entity types (e.g., media content, user profile images) to improve detection?
• What multimodal extensions (image, video, audio) could enhance bot detection for multimedia bots?