Birdwatch: Crowd Wisdom and Bridging Algorithms can Inform Understanding and Reduce the Spread of Misinformation

Contributions

• Algorithm using matrix factorization to identify crowdsourced annotations that are both subjectively informative and broadly persuasive across heterogeneous populations
• Introduction of "bridging-based" ranking to surface notes that appeal across political lines, reducing echo-chamber effects
• Large-scale survey evidence (Wave 1, 2, 3 experiments, N=6,775–9,108) demonstrating that algorithm-selected notes improve perceived understanding of issues raised in tweets
• Empirical evidence from randomized A/B tests showing annotations reduce sharing behavior on Twitter (25% lower engagement with notes)
• Open-source release of Birdwatch data, code, and methodology

Method

Birdwatch system: Twitter launched Birdwatch on January 25, 2021 as a pilot program. Contributors can add free-text notes or structured responses to any tweet they view, rating notes as "helpful" or "not helpful." Users see notes displayed below tweets and can access contributor profiles and note-rating timelines.

Data collection:

• Wave 1 (April 2022): 113 notes on contributor-selected tweets; 75 notes rated by raters
• Wave 2 (April 2022): 119 notes on a fixed set of 46 tweets; 38 survey respondents
• Wave 3 (December 2021): 113 notes selected during a 4-week period via algorithm + supermajority voting; final matrix with 7,544 raters, 10,321 tweets
• After filtering (minimum quality thresholds), 5,787 notes included in final analysis

Matrix factorization algorithm: The authors model note-rater interactions as a sparse matrix. For each note and rater, entries encode whether a note is rated "helpful" (1) or "not helpful" (0), with nulls for unrated pairs. The algorithm performs low-rank matrix factorization to learn latent representations:

• User intercept term (μ): global helpfulness propensity
• User-specific term (l_u): user's leniency in rating notes helpful
• Note-specific term (f_n): note's intrinsic appeal
• Interaction term (u_u · f_n): how well a note matches a rater's viewpoint

The algorithm applies L2 regularization and separate biases for rater-user factors, then uses note intercept thresholds to label notes "helpful" or "not helpful." Provisional helpfulness filters apply early (6/7 valid ratings, ≥66% helpful in first 48 hours, rating density requirements).

Survey experiments: In three waves, participants were shown tweets with or without notes and asked whether they found the note helpful for understanding, whether it was accurate, and whether they agreed with the underlying claim. Effects tested via binomial logit models controlling for party ID, age, gender, and race.

A/B engagement test: A/B experiment (August–September 2022) compared users randomly assigned to show/no-show conditions and measured retweet, quote, "like," and reply rates.

Results

Algorithm evaluation (RQ1):

• Notes selected by the algorithm have significantly higher perceived helpfulness than control group (average log odds coefficient p < 0.001)
• Effect robust across Democrats and Republicans but larger for Democrats (Table 6)
• Supermajority voting baseline underperformed the matrix factorization algorithm on precision

Perceived accuracy and claim agreement (RQ2):

• Modest correlation (r = 0.45) between perceived accuracy and agreement with the Tweet in control group
• Exposure to algorithm-selected notes weakly attenuates this correlation (r = 0.29), suggesting annotations do help users decouple accuracy from agreement
• Perceived accuracy alone does not reliably predict agreement; the "notes are most helpful when perceived as accurate" framing is incomplete

Spread reduction (RQ3):

• A/B test: users who see annotations are 25–34% less likely to retweet, quote, or reply to annotated tweets
• Effect is consistent across engagement types (retweet, quote, like, reply)
• Suggests exposure to high-quality, cross-partisan annotations does reduce behavioral amplification

Bridging and polarization (Key finding):

• Notes identified as appealing across political lines ("broadly helpful notes" in the intercept distribution, Figure 2) do achieve higher perceived helpfulness among both Democrats and Republicans
• Largest effect on users who initially disagree with the claim (Figure 5, rightmost panels)
• Widening diversity of contributor base from US to international users and the bridging algorithm's success support possibility of reducing polarized echo-chamber effects

Notes

Strengths:

• Real-world deployment on a major platform with millions of users and genuine crowdsourced data
• Methodologically sound use of matrix factorization for heterogeneous treatment effects across political affiliations
• Multiple evaluation waves with successively larger and more representative samples
• A/B test results on engagement are novel and evidence-based; most prior fact-checking work relies on self-reported intent
• Open-source code and data release enabling replication and follow-up work
• Clear framing of two distinct challenges: (1) identifying notes that inform understanding vs. (2) notes that reduce polarization

Limitations:

• Survey experiments rely on self-reported helpfulness and accuracy perceptions; actual belief change not measured
• A/B test uses only behavioral engagement (likes, retweets, replies), not downstream belief adoption or misinformation reduced
• Algorithm thresholds for "helpful" labels are data-driven but somewhat arbitrary (e.g., 0.40 and −0.08 thresholds for intercept); sensitivity analysis limited
• Notes on political topics may face higher bias from partisan raters; weighting and filtering mitigate but don't eliminate
• Birdwatch is opt-in and skews toward more engaged, politically aware users; generalizability to broad population unclear
• Paper does not address potential for malicious actors to game the rating system with coordinated false votes

Open questions:

• Does exposure to bridging notes durably change beliefs or just immediate behavior?
• How does algorithm performance scale as the contributor pool diversifies beyond the US?
• What makes certain notes bridge effectively? Is it neutrality, additional context, or acknowledgment of trade-offs?
• Can the approach be applied to domains beyond politics (e.g., health, science) where consensus is higher but polarization less salient?