Political motives help rather than hinder crowdsourced fact-checking

DOI/URL: Open Science Framework

Contributions

• Theoretical framework: political and accuracy motives are independent rather than antagonistic; both can drive content flagging behavior
• Empirical evidence from two complementary datasets: controlled survey + field data from X's Community Notes program
• Demonstration that political motivation predicts higher flagging quantity without sacrificing discrimination quality
• Finding that politically motivated participants particularly excel at identifying politically discordant false content
• System-level evidence that politically motivated contributors benefit crowdsourced fact-checking at scale

Method

The authors conducted a preregistered survey study on Lucid (N=2,380, representative sample) where participants were shown a simulated news feed with 80 headlines—60 political (balanced on veracity and partisan lean) and 20 non-political entertainment posts. Participants flagged posts they believed false or misleading and provided written explanations.

To measure political motivation, they assessed out-party animosity, issue polarization, and political knowledge via principal component analysis. The authors tested predictions using logistic and ordinary least squares regressions, controlling for demographics and education.

For field validation, they analyzed X's Community Notes dataset (4,025 users, 3,151 unique flagged tweets, 4,442 tweet-notes) created between January–June 2021. They sampled 461 tweets with professional fact-checker evaluations and classified notes as concordant or discordant based on partisan alignment with the flagging user. Notes were evaluated for accuracy and partisanship.

Results

Survey findings:

• Participants with higher political motivation contributed roughly double the number of overall flags (6.54 vs. 10.08 flags).
• More politically motivated users flagged a significantly greater proportion of politically discordant false content (79.1% agreement with professional fact-checker ratings).
• Highly politically motivated individuals were substantially less biased than less motivated participants when comparing discrimination rates (ability to distinguish true from false): both groups showed strong discriminative ability.
• Participants flagged more posts overall as political motivation increased, independent of accuracy motivation (quasi-Poisson and OLS regressions: b≈0.23–0.24, p<.001).

Community Notes field data:

• More politically extreme users contributed significantly more notes overall and more notes on misidentified and false content.
• 79.1% of Community Notes classifying tweets as misled agreed with professional fact-checkers' ratings.
• Politically extreme users maintained high classification discrimination even when writing more flagging notes on politically discordant content.
• Bridging algorithm scores (ratings other users gave to notes) showed no significant disadvantage for notes written by politically extreme users; political extremism did not correlate with lower helpfulness ratings.
• System-level: excluding politically extreme contributors would reduce total fact-checking volume by roughly one-third without improving accuracy.

Notes

Strengths: The paper makes a counter-intuitive and well-supported argument backed by both controlled experiments and real-world platform data. The theoretical framing—that political and accuracy motives operate independently—is parsimonious and testable. Field validation on X's Community Notes is particularly valuable given its scale and real stakes. The writing is clear and the statistical analyses are rigorous with pre-registration.

Limitations: The survey task is more artificial than real-world scrolling (participants see curated feeds and know their judgments are being studied). Political context in the US may not generalize to other democracies or non-Western platforms. The paper focuses on flagging behavior but doesn't examine whether downstream interventions (warnings, downranking) are equally effective when driven by political vs. accuracy motives. Community Notes is a specific platform with its own norms and user base.

Open questions: How do political motives interact with system design (e.g., do algorithms amplify or dampen politically motivated flagging)? Can platforms harness political motivation without amplifying echo chambers? Does the effectiveness of politically motivated fact-checking vary by content domain (elections vs. health vs. science)?