Perceived legitimacy of layperson and expert content moderators

Cameron Martel; Adam Berinsky; David Rand; Amy X. Zhang; Paul Resnick

In brief

A nationally representative survey of 3,000 U.S. participants evaluates public perceptions of legitimacy for different content moderation approaches. Expert juries (domain specialists, fact-checkers, journalists) are perceived as most legitimate, but laypeople with increased size, knowledge qualifications, political representativeness, and allowed discussion can achieve comparable legitimacy. Partisan divides exist but can be bridged through jury design choices; algorithms are perceived as least legitimate.

Contributions

Empirically evaluates Americans' preferences for competing content moderation approaches (expert juries, layperson juries, algorithms, coin flips) using a nationally representative conjoint survey (N=3,000).
Identifies specific jury composition and procedural features that increase layperson legitimacy: size (30 vs. 3 members), knowledge qualifications, discussion allowance, and political balance.
Documents partisan asymmetries: Republicans perceive experts as significantly less legitimate than Democrats, yet still more legitimate than baseline layperson juries.
Demonstrates that legitimacy perception is a key design consideration for platform moderation systems; shows features can be manipulated to bridge political divides.

Method

Participants completed a conjoint survey with rank-ordered preference questions about hypothetical content moderation juries. Each jury varied on four features:

Jury composition: Three layperson jury types (nationally representative, politically balanced, or social media users), three expert jury types (fact-checkers, journalists, or domain experts), and two non-jury baselines (coin flip, algorithm).

Size: 3, 30, or 3,000 members.

Qualifications: None, or demonstrated minimum news knowledge and reasoning ability.

Discussion: Independent evaluation or allowed deliberation during evaluation.

Participants evaluated 20 juries across 10 pairs, assessing legitimacy on a 7-point Likert scale adapted from prior work on procedural justice. A pairwise design asked participants to choose which jury configuration they'd prefer for content moderation. Analyses used linear regression predicting legitimacy ratings and logistic regression predicting jury choice, with preregistered specifications and robustness checks.

Results

Baseline legitimacy: Domain experts receive the highest baseline legitimacy ratings (4.01 on 1–7 scale), significantly higher than fact-checkers (3.78) and journalists (3.21). Layperson juries receive substantially lower baseline ratings (3.40 for nationally representative, 3.30 for politically balanced, 3.00 for social media users). Non-jury baselines are lowest: coin flip (2.78), algorithm (3.00), CEO (1.39).

Jury composition effects increase layperson legitimacy: All three layperson jury types showed significantly increased legitimacy when:

Size increased from 3 to 30 (p < .001); minimal additional benefit from 30 to 3,000 (p > .05).
Minimum knowledge qualifications required (p < .001).
Discussion allowed (p < .001).

Additive effects: The combined effects of size (30), qualifications, and discussion for layperson juries approach or exceed baseline expert legitimacy. Nationally representative and politically balanced juries with maximal features (size 30, qualifications, discussion) achieve legitimacy ratings comparable to domain expert panels.

Partisan asymmetries: Republicans perceive domain experts as significantly less legitimate than Democrats (p < .001), though Republicans still rate experts more legitimate than layperson juries at baseline. Increasing jury size and adding qualifications significantly increase layperson jury legitimacy for Republicans (p < .001); allowing discussion for politicians balanced juries further closes the partisan gap (p = .002). However, Republicans never quite reach Democratic-level legitimacy for expert panels.

Pairwise jury choice: In paired comparisons, maximal layperson juries (nationally representative, size 30, qualifications, discussion) match or exceed choice probability of expert juries, with highest-performing expert panel (domain experts with discussion): choice probability 82% vs. 57% for best layperson configuration.

Notes

Strength: Large nationally representative sample with careful experimental design (conjoint methodology) allows for fine-grained analysis of which jury features matter for legitimacy. Pre-registration and multiple robustness checks enhance credibility.

Limitations: Survey measures hypothetical legitimacy perceptions rather than actual behavior or outcomes; layperson juries with maximum features achieve parity on legitimacy but actual moderation accuracy unknown. Study specific to U.S. context and Facebook framing; cross-national and cross-platform generalization unclear.

Significance: Provides empirical grounding for platform design choices in content moderation governance. Evidence that layperson juries can be perceived as legitimate as experts with appropriate procedural design suggests potential for scaling community-based moderation, though implementation challenges remain.

How it connects

Builds on Will the crowd game the algorithm? Using layperson judgments to combat misinformation on social media by downranking distrusted sourcesEpstein et al. · OSF Preprints · 2019 Related Partisan consensus and divisions on content moderation of misinformationMartel et al. · PsyArXiv · 2024 Related Human Cooperation and the Crises of Climate Change, COVID-19, and MisinformationVan Lange & Rand · Annual Review of Psychology · 2022 Related Accuracy prompts protect professional content moderators from the illusory truth effectLin et al. · PNAS Nexus · 2024