Skip to content

Policy violation detection

Policy violation detection addresses the problem of identifying content that violates explicit platform policies or community rules. Unlike generic harmful-content detection (which aims to identify content that is offensive, hateful, or toxic regardless of policy context), policy-violation detection must correctly map content to a specific rule or policy and determine whether the rule applies. This requires understanding both the content and the operational context (which community, which policy, etc.).

Key differences from harmful-content detection

  • Rule-grounded: Violations are defined relative to explicit textual policies, not universal notions of harm.
  • Context-dependent: The same content may violate one policy in one community but comply with rules in another.
  • Structured output: Decision must identify which rule is violated, not just whether violation exists.
  • Accountability-oriented: Decisions affect user rights (removal, suspension, appeal) and require justification.

Key papers

  • Benchmarking System One Models in Online Moderation — Benchmarks System One Models for policy-violation detection across PluRule and HateModerate; evaluates impact of rule inventories, retrieved precedents, and retrieval-augmented decisions on exact policy selection accuracy; shows precedent retrieval improves Jev by +12–22 EMA points