Field experiments on social media

Mohsen Mosleh; Gordon Pennycook; David G. Rand

doi:10.31234/osf.io/dgmc2

In brief

Online social media data enables unprecedented observational insight into human behavior, but causal inference requires randomization. This review surveys experimental approaches combining field experiments with social media: hybrid lab-field designs (randomizing exposure to content and measuring effects via surveys and observed online behavior) and pure field experiments (randomizing treatments administered directly to users without their knowledge). Examples include studies of partisan polarization, misinformation correction, fact-checking interventions, and harassment reduction. The paper discusses methodological tradeoffs and ethical constraints.

Contributions

Comprehensive methodological review of field experiments conducted on social media platforms
Distinction between hybrid lab-field studies (randomizing exposure, measuring via surveys and digital traces) and pure field experiments (randomizing treatments in-situ with observed behavioral outcomes)
Examples spanning political psychology, misinformation intervention, fact-checking, and content moderation
Practical guidance on study design, participant recruitment, and analytic approaches
Detailed discussion of ethical considerations: privacy, informed consent, minimal risk, and reproducibility challenges

Method

The paper reviews recent methodological innovations in experimental social media research. Hybrid lab-field designs recruit participants with social media accounts, randomize exposure to content (e.g., requiring them to follow opposing political accounts for a period), and measure impacts through follow-up surveys and observed digital traces. Pure field experiments randomize treatments directly in the platform's production environment—via private messages, public posts, or algorithm-driven delivery—observing effects on subsequent user behavior without participants knowing they are in an experiment. This combination preserves causal inference while maintaining ecological validity and avoiding experimenter demand effects.

Examples include: randomizing exposure to partisan Twitter accounts and measuring changes in political polarization; delivering private fact-checking messages to users who shared misinformation, measuring downstream sharing behavior; randomizing sanctioning of racist speech and measuring reductions in future harassment; and using survey experiments within social media platforms to measure misinformation susceptibility while capturing actual online behavior in baseline and subsequent waves.

Results

The paper presents qualitative evidence from multiple published studies:

Partisan exposure experiments (Bail et al. 2018): exposing Republicans to diverse Twitter content increased polarization; exposing to oppositional partisan accounts had smaller effects.
Fact-checking interventions (Pennycook et al. 2021): private messages priming accuracy increased quality of subsequent news shared; effect persisted 24 hours post-intervention and varied by news outlet trustworthiness.
Social correction (Mosleh et al. 2021c): public corrections to false news by researcher accounts increased subsequent sharing of low-quality partisan and toxic content—a perverse downstream consequence.
Deactivation studies (Allcott et al. 2020): randomly deactivating Facebook increased well-being but reduced news knowledge.
Harassment reduction (Munger 2017): accounts with high follower counts and strong racial cues reduced racist speech when sanctioning it.

Notes

This is a high-impact methodological review from the Rand/Pennycook lab, now a canonical reference for field experimental approaches to online behavior. The paper's strength is its breadth—it captures the nascent field of field experiments on social media and provides practical guidance alongside ethical cautions.

Key limitations noted by the authors: (1) survey-based studies measure sharing intentions not actual sharing, and may differ in important ways; (2) field experiment stimuli (researcher-created posts) may differ from organic content users naturally encounter; (3) treatment delivery via platform mechanisms (ads, algorithm) is opaque and subject to platform changes; (4) ethical tensions between ecological validity (not disclosing experiments) and informed consent, solved via IRB exemptions but raising reproducibility concerns; (5) social media platforms change rapidly, making replication difficult. The authors argue that positive interventions (aimed at reducing harmful outcomes) should be prioritized over studies that might increase misinformation or harassment, even for scientific purposes.

The paper is methodologically sound and practically useful for researchers planning field experiments; the ethical framing is balanced and thoughtful. It does not itself report novel empirical findings but instead synthesizes existing work—the value lies in the taxonomy and guidance it provides to the community.