Key claims
- Botometer scores are imprecise: ROC-AUC of 0.85 on the authors' combined dataset (vs. 0.94 claimed in the original Botometer paper), but more critically, precision-recall analysis shows false-positive rates are unacceptably high in realistic bot/human ratios.
- Thresholds are unreliable: At a threshold of 0.76 (used in published studies), Botometer classifies 59% of accounts as bots in a realistic population (15% actual bots), but 41% of those are false positives — humans wrongly flagged as bots.
- Language matters: Botometer performs significantly worse on non-English tweets (German accounts showed ROC-AUC of 0.76 vs. 0.93 for US politicians).
- Temporal instability: 27% of bot accounts and 7% of human accounts cross the 0.76 threshold on different days during a 3-month observation window, making Botometer snapshots non-reproducible.
- Population imbalance is ignored: Studies use Botometer's raw thresholds without accounting for the fact that humans vastly outnumber bots on Twitter — a small false-positive rate on the training set becomes a huge false-positive rate in practice.
Sources cited in the article
The paper cites foundational bot detection work including:
- Varol et al.'s original Botometer paper (creator's labeled dataset)
- Cresci et al.'s work on spambot evolution and detection limitations
- Grimme et al.'s experiments showing Botometer struggles with hybrid/automated accounts
- Fernquist et al., Echeverría et al. — studies showing Botometer struggles with non-English data and specific bot types
Notes
Strengths: Rigorous empirical evaluation using multiple datasets (politicians, creative bots, Botometer's own training data). Uses appropriate statistical methods (precision-recall curves, ROC-AUC, temporal analysis) and clearly explains why ROC-AUC alone is misleading for imbalanced populations. The authors transparently acknowledge Botometer's legitimate uses but honestly describe its limitations.
Why it matters: This 2020 paper documents a critical gap between how Botometer is trained (on balanced datasets) and how it's applied (to Twitter's imbalanced population of mostly humans). Hundreds of published papers have used Botometer thresholds without validation, potentially inflating bot prevalence estimates. The temporal instability finding is particularly damaging — results from a single day are not reproducible.
Limitations: The study tested Botometer v3 (released 2018); Botometer v4 has since been released with improvements, though the authors note major issues (language coverage, temporal stability, lack of clarity on bot types) remain unaddressed. The German bot dataset was small (27 accounts), limiting generalizability of the language findings.
Research gap: The paper strongly advocates for human validation and alternative detection methods (group-level unsupervised classifiers, forensic analysis) but does not propose a practical replacement for researchers analyzing large-scale Twitter data. The tension between scale and validity remains unresolved.