Content moderation

Content moderation encompasses policies, enforcement mechanisms, and governance structures platforms use to manage harmful, false, or violative content. This includes both moderation of human-generated content and, increasingly, governance of AI-generated content and AI-assisted moderation tools.

Key approaches

Detection-based: Identifying and removing harmful content through human review, crowdsourcing, or algorithmic detection

Prevention-based: Structural design choices that reduce the spread of false content (e.g., algorithmic transparency, friction, diversification)

Governance-based: Establishing policies, appeals processes, and external review mechanisms

Key videos

• Rand — How Polarization May Help Combat Misinformation — Explores scaling content moderation through crowd-sourced fact-checking systems; examines how community-based approaches can leverage diverse perspectives to identify false and misleading content

Key papers

• VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation — VEX-Bench: reveals asymmetry between generation cost and verification cost for LLM-generated misinformation; shows that high-VEX content is often prioritized during moderation screening despite expensive verification cost, introducing systematic risk of misallocation in resource-constrained systems.
• Two Fault Lines: Latent Polarity Geometry in X Community Notes — Empirical analysis of X's Community Notes crowdsourced fact-checking system; shows disagreement is two-dimensional, with hidden topic-based axes driving publication failures
• Modeling Duelling Contagions of True and False Information in the Face of Inherent — Counterfactual analysis of moderation-relevant intervention strategies: strategic seeding with self-censoring agents and deployment of well-informed external agents to counter misinformation dominance
• One AI Signal, Many Human Judgments: A Bayesian Cascade Analysis of AI-based Credibility Indicators in Online Information Spread — Analyzes AI-based credibility indicators as a moderation tool; shows how AI signals reshape user beliefs and public consensus via information cascades; identifies when AI interventions help vs. harm, and how structural interventions (diversification, expert review) can improve outcomes

Platform governance and regulation

• Control-Theoretic Content Moderation — Control-theoretic framework for composing moderation decisions into effective platform-level strategies; introduces proportionality constraints and multi-objective optimization (MPC and PID-based moderators) and demonstrates they outperform fixed-rule baselines in balancing harmfulness, engagement, and activity
• Misinformation, Content Moderation, and Information Integrity — Wardle examines the trilemma of content moderation: lack of clear legal authority to decide what content is harmful, impossible scale (millions of uploads per second), and conflicting interests across platforms, governments, and civil society; proposes crowdsourced infrastructure, data repositories, and coordinated networks as solutions
• Disinformation and Platform Governance: Moderation, Alt-Tech, and Radicalization — Interview with Kate Starboard on disinformation, platform moderation, the migration of harmful activity to alt-tech platforms, and the absence of regulatory frameworks governing content moderation
• The platform governance triangle: conceptualising the informal regulation — analyzes informal multi-stakeholder governance arrangements for platform content, using the governance triangle model to examine Facebook's Oversight Body and similar initiatives

Omission-based misinformation detection and mitigation

• What's Left Unsaid? Detecting and Correcting Misleading Omissions in Multimodal — Detection and correction of misleading omissions in news previews (headline+image); proposes interpretation-aware fine-tuning for detecting omission-based mismatch and rationale-guided correction; shows most cases require missing context addition, and image-driven cases require multimodal interventions

AI-generated content detection and mitigation

• Drowning in AI Slop: How Social Media Platforms (Do Not) Label AI and Deepfake Content under EU law — Empirical audit of platform labeling practices for AI-generated and deepfake content under EU Digital Services Act; identifies major implementation gaps—only 34–82% of posts labeled across platforms—with worst performance in systematic-risk contexts (elections, civic discourse, security).
• Making AI-Assisted Claims Independently Challengeable: Publication Authority and a Protocol for Falsifiable Publication Records — Publication Authority protocol ensuring AI-assisted claims are independently verifiable, falsifiable, and challengeable through formal binding of evidence, authorization, measurement, and lifecycle events
• Counter-GEO-Bench: Evaluating Defenses Against Information-Distorting Generative Engine Optimization — Counter-GEO-Bench: evaluates guardrails against misinformation injection in LLM synthesis pipelines; shows existing chunk-level and output-rail filters fail against information-distorting GEO attacks; proposes C-GEO Guard contrastive detector achieving 47.6% relative ASR reduction
• Your AI Text is not Mine'': Redefining and Evaluating AI-generated Text — Systematically analyzes fragmented task definitions in AI-generated text detection (AITD); proposes unified framework distinguishing different notions with explicit assumptions about what constitutes "AI text"; introduces AITDNA dataset of real human-AI co-creation; shows detector performance depends on alignment with implicit normative standards
• Industrialized Deception: The Collateral Effects of LLM-Generated Misinformation — Comprehensive synthesis of mitigation strategies including LLM-based detection, adversarially-aware training, inoculation and prebunking approaches, provenance infrastructure (C2PA), and platform design interventions. Emphasizes that purely technical countermeasures face significant challenges; recommends infrastructure-level interventions including content authenticity standards and algorithmic transparency.
• Generative Language Models and Automated Influence Operations: Emerging Threats — mitigation strategies for detecting and slowing the spread of AI-generated propaganda; discusses challenges in identifying synthetic text and platform-level coordination requirements

Offensive and abusive content detection

• Toxicity in ChatGPT: Analyzing Persona-assigned Language Models — analysis of toxic output generation by ChatGPT under persona assignment; identifies a safety vulnerability and discriminatory bias in AI-generated content
• Comparative Studies of Detecting Abusive Language on Twitter — Benchmarks detection models for abusive language on Twitter; establishes baseline performance for automated content moderation systems
• Predicting the Type and Target of Offensive Posts in Social Media — Detection and hierarchical classification of offensive content
• SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social — Benchmark evaluation of offensive content detection and categorization systems across 115 teams; demonstrates feasibility of hierarchical detection, type categorization, and target identification
• Mohseni & Ragan (2018) — Combating Fake News with Interpretable News Feed Algorithms — Argues that transparent news feed algorithm design is a prevention-based approach complementary to detection
• Truthful AI: Developing and Governing AI That Does Not Lie — governance frameworks for AI truthfulness as applied to platform content and AI systems
• Lazer et al. (2018) — The Science of Fake News — challenges in detection and moderation

See also