Causally estimating the effect of YouTube’s recommender system using counterfactual bots

Homa Hosseinmardi; Amir Ghasemian; Miguel Rivera-Lanas; Manoel Horta Ribeiro; Robert West; Duncan J. Watts

doi:10.1073/pnas.2313377121

In brief

Using "counterfactual bots"—programmatic accounts trained on real user histories but following rule-based viewing patterns—the authors estimate YouTube's causal effect on partisan content consumption. They find that algorithmic recommendations, on average, moderate a user's experience relative to their own preferences, with the effect strongest for heavy partisan consumers. The results suggest that user preferences, not algorithmic bias, primarily drive polarized consumption on YouTube.

Contributions

Novel counterfactual bot methodology to isolate algorithmic effects from user preferences by comparing bots following user histories vs. bots following rule-based trajectories (e.g., "up next" recommendations).
Causal evidence that YouTube recommendations produce less partisan consumption than what users would watch following their own intrinsic preferences, especially for extreme users.
Evidence that recommendations "forget" partisan preferences quickly: sidebar recommendations converge to neutral within ~30 videos after a user switches to moderate content, though homepage recommendations adjust more gradually and are influenced by prior history length.
Empirical resolution of conflicting prior work: panel studies (showing user preferences dominate) and audit studies (showing algorithmic bias) were asking different questions; this work directly compares counterfactuals.

Method

Experimental design: Bots trained on real user viewing histories from October 2021–December 2022 (87,988 US desktop panelists, 4,583 heavy YouTube consumers). Each bot experiment has two phases:

Learning phase: All bots watch the same sequence (real user history) to train the recommender.
Observation phase: Control bot continues following the real user; counterfactual bots follow rules (e.g., click "up next" sidebar recommendation, click random sidebar video, or random homepage video).

By comparing control vs. counterfactual bot behavior, the authors measure user preference (difference between what the user would watch vs. what the recommender promotes).

Partisanship scoring: Videos assigned partisan scores at the channel level using embeddings from reference 25, validated against crowdsourced labels (80%+ agreement). Categories: far-left (fL), left (L), center (C), right (R), far-right (fR).

Sample: Stratified to oversample heavy far-right consumers (1.12% of histories vs. 66% centrist), yielding 125 focal users for the bias experiment and 44 for the "forgetting time" experiment.

Results

Main finding: algorithmic moderation, not amplification.

For all three rule-based counterfactual strategies (up next, random sidebar, random homepage), the control bot—following the real user's preferences—consumed more partisan content than the counterfactual bots. This gap widened with user extremism (Figure 3). For centrist users, the effect was small; for far-right-heavy users, algorithmic recommendations produced substantially less partisan content. Linear regression confirmed: "preference" (user choice over algorithm) was positive and significant for all recommendation types, indicating that users prefer more extreme content than the algorithm recommends.

Bursty consumption: Users who consumed "bursts" of far-right videos (2–6 in succession) subsequently watched more partisan content, but so did identical counterfactual bots following recommendations after the same burst. The effect was not due to algorithmic amplification but to users signaling a preference shift.

Forgetting time: When a bot switched from far-right to centrist consumption:

Sidebar: Far-right content disappeared within ~30 videos; convergence was rapid and unaffected by prior history length.
Homepage: Gradual adjustment; even after 90 videos, some far-right recommendations remained. Longer prior histories of far-right consumption slowed homepage convergence but did not affect sidebar forgetting.

Robustness: Results held when using "establishment score" (anti-establishment positioning) instead of partisanship. Far-left consumers showed similar patterns (not amplified by recommender). Effects persisted when bots subscribed to most-watched channels.

Notes

Strengths:

Novel, rigorous counterfactual design with high ecological validity (trained on real user behavior, tested on live YouTube).
Large, representative sample from Nielsen panel; oversampling of rare heavy-partisan consumers.
Directly addresses the unresolved tension between panel studies (user preferences dominate) and audit studies (algorithms bias toward extremes).
Multiple robustness checks (different scoring metrics, far-left users, subscriptions).

Limitations:

Findings apply only to post-2019 YouTube algorithm (major changes in 2019). Pre-2019 radicalization concerns may have been valid.
Desktop-only data; mobile consumption patterns may differ.
Bot viewing accelerated to simulate months in 2–3 days; unclear if slow real-time viewing would yield different recommender responses.
Channel-level partisanship scoring; within-channel variation ignored.
Data collected Oct 2021–Dec 2022; experiments run early 2023. Algorithm may have drifted since.

Implications: The paper redirects concern from algorithmic manipulation toward structural factors: supply and demand of extreme content, user preferences, off-platform radicalization pathways. This does not absolve platforms of responsibility but clarifies where intervention should target.