Birds of a feather don't fact-check each other: Partisanship and the evaluation of news in Twitter's Birdwatch crowdsourced fact-checking program

Jennifer Allen; Cameron Martel; David G. Rand

doi:10.1145/3491102.3502040

In brief

Partisanship strongly shapes how Birdwatch users evaluate tweets and fact-checks. Users are more likely to flag counter-partisan tweets as misleading and to rate co-partisan fact-checks as helpful, even after controlling for content features. These partisan patterns are highly predictive of both classifications and helpfulness ratings, suggesting that crowdsourced fact-checking platforms must account for partisan biases when aggregating community judgments.

Contributions

Directly measures partisan bias in crowdsourced fact-checking by analyzing 4,910 fact-check notes and 28,276 helpfulness ratings from Twitter's Birdwatch program
Quantifies the relationship between shared partisanship (tweeter-writer, writer-rater) and both misleadingness classification and helpfulness evaluation
Shows that partisanship features are more predictive than content features alone, with note-writer and tweeter partisanship among the top predictors
Demonstrates asymmetric partisan patterns: Republicans rated 97.2% of their own tweets as misleading, Democrats 82.4%, both far above baseline

Method

Used Birdwatch's public dataset covering January–June 2021, comprising three linked datasets: Notes (4,910 notes by 1,092 Birdwatchers), Ratings (28,276 helpfulness ratings by 2,359 raters), and Tweets (full text and metadata for tweets annotated). Inferred user partisanship using M3Model machine-learning approach applied to Twitter profile images and text. Built random forest (RF) models to predict: (1) whether a note classified its tweet as "Potentially Misleading" or "Not Misleading"; (2) whether a rating judged a note as "Helpful" or "Not Helpful." Evaluated using Area-Under-the-Receiver-Operating-Curve (AUC) with 5-fold cross-validation. Extracted features from three categories: content (length, sentiment, readability, URLs), context (followers, statuses, age, gender, partisanship), and combinations thereof. Conducted logistic regression to examine directional relationships between key features.

Results

Misleadingness classification: Content features alone achieved AUC = 0.56 (baseline ~0.5); adding partisanship scores increased AUC to 0.84 (range 0.77–0.89); adding all demographic/engagement context features yielded AUC = 0.87 (0.80–0.92). Note-writer partisanship and tweeter partisanship were among the top-three most important features. Logistic regression revealed a negative interaction between tweeter and note-writer partisanship (b=-1.25, p<.001): when shared partisanship is high, users are less likely to rate notes as accurately identifying misleadingness.

Helpfulness classification: Content-only model (AUC = 0.76, range 0.74–0.77) substantially underperformed partisanship-only model (AUC = 0.89, range 0.88–0.90). Feature importance analysis showed rater partisanship score as the single largest predictor (0.21), followed by note-writer partisanship (0.10). Logistic regression found a positive interaction between note-writer and rater partisanship (b=1.27, p<.001): shared partisanship strongly predicts helpfulness. Democrats were 3X more likely to rate Democratic-authored notes as helpful; Republicans 1.5X more likely to rate Republican notes as helpful.

Partisan asymmetries: Predicted Republicans rated 97.2% of tweets from predicted Republicans as misleading; predicted Democrats rated 82.4% from predicted Democrats as misleading—both far exceeding the opposite-party baseline (Republicans: 82.4% of Democratic tweets; Democrats: 71.3% of Republican tweets).

Notes

Strengths: Unprecedented direct access to real crowdsourced fact-checking judgments with rich user behavioral data. Large sample (4,910 notes, 2,359 raters). Systematic comparison of feature importance across content, context, and partisan dimensions. Clear methodological separation of RF (predictive power) and logistic regression (interpretation).

Limitations: Birdwatch pilot participants were self-selected and unrepresentative—more politically engaged, more male (81% in 5,609 raters), average age skewed older. Data collected Jan–Jun 2021 before Birdwatch renamed to Community Notes and changed helpfulness-aggregation algorithm (June 2021, weighting by rater reputation). Authors note they cannot determine whether patterns reflect prior factual beliefs or active partisan motivated reasoning. Partisan inference relies on M3Model, which has known demographic biases.

Implications: Partisan herding in crowdsourced systems is not inherent to the crowdsourcing mechanism itself but driven by user identity salience. Raises important questions about aggregation design: notes rated helpful by co-partisans may receive reputation boosts that inflate their visibility, creating a "cheerleading effect." The paper does not claim crowdsourcing is ineffective, but highlights the importance of designing aggregation and visibility rules that account for partisan dynamics.