Measuring Misinformation and Conspiracy Theories on YouTube's Search and Recommendation Algorithm

Summary

Mitra presents a systematic audit of YouTube's search and recommendation algorithms to assess whether they surface conspiracy theories and to understand the effect of user demographics, watch history, and topic on content recommendation. Using scripted Selenium bots emulating users with different personas, the team conducted search audits (fresh accounts, no history) and watch audits (with prior consumption of conspiratorial, neutral, or debunking videos) across five conspiracy topics: 9/11, vaccine, moon landing, chemtrails, and flat earth. Key findings: demographics alone (age, gender, geolocation) show no significant effect on fresh-search results, but watch history is highly influential—especially a gender effect where male accounts receive more misinformation recommendations even when consuming neutral content. YouTube's response is topic-dependent and reactive: vaccine recommendation algorithms show improvement (more debunking suggestions), but chemtrails and 9/11 remain problematic. This reveals YouTube's handling of misinformation is inconsistent across topics.

Key research questions

1. Do user demographics (age, gender, geolocation) affect the amount of misinformation YouTube returns?
2. Does watch history (prior consumption of pro-conspiracy, debunking, or neutral videos) change recommendation behavior?
3. Does the effect differ by conspiracy topic?

Methodology

Audit framework: Systematic experimentation using:

• Search audits — New accounts with no prior watch history, firing queries representing each conspiracy topic
• Watch audits — Accounts with curated watch history (20 videos of pro-conspiracy, debunking, or neutral stance), then querying and observing recommendations
• Control for noise — All accounts used Firefox in incognito mode, same OS, same timestamp, to isolate the effect of demographics and history from infrastructure/temporal noise

Topic selection: Five conspiracy theories selected via Wikipedia's curated conspiracy-theory list and validated by Google Trends to confirm persistent search interest over 2016–2018.

Query selection: YouTube's auto-suggestions + Google Trends related queries, yielding high-impact, trending search terms for each topic.

Components audited: YouTube's search results and recommendation algorithms (top-5 and "up next" recommendations).

Data collection: 56,000 videos (~2,300 unique) manually annotated as promoting conspiracy, neutral (factual but non-conspiratorial), or debunking.

Findings

Demographic effects (search audit): No significant difference in misinformation prevalence across age, gender, or geolocation for fresh accounts.

Watch-history effects:

• Male accounts receiving higher rates of pro-conspiracy recommendations, even when consuming neutral content (especially for flat earth and vaccine topics) — a concerning finding because these users have not yet formed strong conspiratorial beliefs.
• Vaccine topic: Accounts that watched anti-vaccine videos received more anti-vaccine search results; however, YouTube's recommendation algorithm (top-5, up-next) showed the opposite behavior—recommending more debunking videos. This suggests YouTube has selectively improved the recommendation algorithm for vaccine content but not the search algorithm, creating a filter bubble in search.
• Chemtrails: Accounts watching pro-chemtrails videos received more pro-chemtrails recommendations (both search and recommendations), indicating YouTube has not yet addressed this topic.
• 9/11: Significantly more misinformation in both search and recommendations compared to other topics.

Implications

YouTube's approach to misinformation is reactive rather than proactive:

• Fixes are topic-selective, targeting controversies highlighted by media and critics (e.g., vaccines) while neglecting others (e.g., chemtrails, 9/11).
• Even within a fixed topic (vaccines), fixes are inconsistent: recommendations improved but search filter bubbles persist.
• The gender effect in recommendations (especially affecting users consuming neutral content) suggests YouTube's algorithms may inadvertently direct some demographics toward conspiracy beliefs before they have developed strong conspiratorial worldviews.
• The methodology itself is a contribution — systematic auditing of recommendation algorithms can be adapted to assess other platforms and uncover vulnerable populations.

Notes

This talk provides rare empirical evidence of YouTube's algorithm behavior through black-box auditing, a methodology that remains invaluable given platforms' opacity. The gender effect is particularly striking: users are not self-selecting into misinformation based on strong prior beliefs, but rather being recommended it even from a neutral starting point. The reactive nature of YouTube's fixes (vaccine addressed, chemtrails ignored) underscores that platforms' strategies are driven by external pressure and reputational risk rather than by a systematic principle.