Contributions
- Largest-scale Twitter bot detection dataset: 229,573 users (prior largest: 50,538), with 33.5M tweets and 455K follow relationships.
- Multi-modal user information: First publicly available bot detection dataset to include semantic (tweet content), property (user metadata), and neighborhood (follower/following graphs) information.
- User diversity: Breadth-first search starting from seed users in politics, business, entertainment, and sports ensures geographic and topical diversity, avoiding prior datasets' focus on single scenarios or hashtags.
- Rigorous annotation: Five-annotator crowdsourcing with quality control, verification checks, and manual review of disputed cases to ensure annotation trustworthiness.
- Comprehensive benchmark evaluation: Reproduces 8 competitive bot detection methods and shows systematic performance degradation on TwiBot-20, revealing that prior claimed accuracies (>98%) do not hold on real-world diverse data.
Method
User Selection: Uses breadth-first search (BFS) from 40 seed users across four domains (10 each from politics, business, entertainment, sports) to expand to layer 3, capturing diverse user clusters. Seeds include politicians, media outlets, corporations, artists, and athletes.
Data Collection: Retrieves semantic information (200 most recent tweets per user), property information (38 metadata features per user from Twitter API), and neighborhood information (followers and followings, creating a directed follow graph).
Annotation Strategy:
- Five annotators per user, guided by criteria: lack of originality/pertinence, automated activity, external link promotion, repeated tweets, irrelevant URLs
- Quality control: only annotators >80% accurate on standard questions are trusted
- Consensus reached via 4-of-5 agreement, Twitter verification check, direct messaging for ambiguous cases, and manual team review for disputed users
Results
Dataset Characteristics:
- 229,573 total users (>4× larger than prior datasets)
- 33,488,192 tweets (6× more than cresci-17)
- 8,723,736 user property items
- 455,958 follow relationships (first dataset to include this)
- Annotation: ~60% of bots have reputation score <0.5 (fewer followers than followings), matching prior theory
- Geographic diversity: users from 179 countries, with significant representation beyond top-2
- Topical diversity: hashtags span politics (#Trump, #RNC2020), sports (#SaintsFC), business, and general interests
Benchmark Performance:
- All tested methods show significant accuracy drops on TwiBot-20 vs. prior datasets (e.g., Yang et al. F1: 0.9893 on cresci-17 → 0.8546 on TwiBot-20)
- Botometer (a widely-used public demo): 97.3% accuracy on cresci-17 → 55.8% on TwiBot-20, suggesting real-world Twitter bots have evolved
- Graph-based methods (Alhosseini et al.) fail to generalize when trained on single interest domains, showing TwiBot-20 captures heterogeneous bot types
- Even training on 20% of TwiBot-20 maintains stable baseline performance, showing dataset size is sufficient for deep learning
Notes
Strengths:
- First dataset to include multi-modal information (tweets, metadata, follow graph) at scale, enabling new community-aware detection methods.
- Rigorous annotation protocol with multi-stage quality control strengthens trust in ground truth labels.
- Comprehensive performance study reveals that bot detection remains a hard problem despite prior claims, motivating further research.
- Clear identification of bot characteristics (reputation scores, screen name likelihood) aligns with prior literature, validating annotation quality.
Limitations:
- Annotation snapshot from July–September 2020; bot tactics evolve, potentially dating the benchmark.
- Authors note bots may have adapted post-2017 to evade feature-engineered detectors (e.g., reduced tweet frequency), potentially biasing this dataset toward such bots.
- No explicit annotation of bot types (spam, political, spam followers, etc.) beyond binary classification, limiting fine-grained analysis.
Open questions:
- How does bot detection performance degrade further with real-world class imbalance and streaming data?
- What novel bot behaviors emerge beyond 2020, and do they require dataset updates?
- Can semi-supervised learning leverage the 455K unsupervised users in the support set effectively?