Benchmarking System One Models in Online Moderation

Authors: Federico Mazzoni, Andrea Failla

Venue: arXiv cs.SI, 2026 — arXiv:2610.07953

TL;DR

This paper evaluates System One Models (SOMs)—models that accept natural-language context but return structured, typed outputs—across five benchmarks for online content moderation. The key finding is that Jev, a specific SOM implementation, is competitive with specialized moderation systems on policy-grounded tasks and matches or exceeds Detoxify on harmful-content detection. Retrieved precedents reliably improve Jev's exact policy selection, while Laya (an alternative SOM) benefits less from precedents and behaves more like a general violation detector.

Contributions

• Controlled evaluation framework separating decision context sources (written policies, retrieved precedents, answer-space restrictions) to isolate their effects on moderation performance.
• Broad SOM benchmark across five public datasets (PluRule, HateModerate, OLID, HateXplain, ModerationBench), covering both policy-violation detection and harmful-content classification.
• Confidence calibration analysis: evaluating whether typed SOM interfaces (Choice, Noul, Score) return meaningful confidence values for selective moderation and human review.
• Architectural comparison of Jev (Google) and Laya (open-source) SOMs to understand when structured outputs help versus hurt moderation workflows.

Method

The paper evaluates two SOM implementations on two complementary moderation tasks:

Policy-violation detection — Given a message and a rule inventory, does the model correctly identify which (if any) rules are violated? Evaluation on PluRule (Reddit community-specific rules) and HateModerate (Facebook hate-speech policies) with both high-support communities and full rule inventories. Metrics: Exact-Match Accuracy (EMA), Violating-Rule Recall (VRR), No-Violation Recall (NVR).

Harmful-content detection — Given only the message (no explicit policy), does the model classify it as harmful or offensive? Evaluation on OLID (offensive language), HateXplain (hateful/offensive/normal), and ModerationBench (multimodal Safe/Unsafe). Metrics: AUROC, F1, binary accuracy.

Information conditions manipulate what context is available: - Direct: message only, no precedents. - NAD (No additional data): complete rule inventory, all rules visible. - RAD (Retrieval-Augmented): five nearest similar precedents retrieved by embedding similarity. - General: all rules or represented rules.

SOM interfaces tested: - Choice: selects one outcome from a finite set, returns probability distribution and separate confidence. - Noul: binary proposition returning probability in [0,1], no separate confidence field. - Score: ordinally-scaled score on an anchor-specified scale.

Results

Policy-violation detection: - Jev-compact with General-all rule context achieves 65.1 EMA on PluRule six-community subset, outperforming Laya (36.7 EMA). Retrieved precedents improve Jev by +12.35 to +22.20 EMA points across datasets. - Laya shows much weaker precedent improvement and operates more like a broad violation detector. - On HateModerate, General-retrieved rules improve both Jev and Laya, but semantic RAD labels reduce Jev's binary accuracy by 5.06 points while increasing Laya's.

Harmful-content detection: - On OLID, Jev Direct achieves 74.4% accuracy, closely matching Detoxify variants (79.7–81.0%). Adding precedents improves AUROC from .789 to .816. - On HateXplain, Jev RAD-semantic reaches 66.1% accuracy and 65.9% F1. Laya shows inconsistency: retrieval degrades discrimination from .749 to .604 AUROC. - On ModerationBench, Jev with RAD-semantic and Choice interface achieves 87.4% accuracy and 79.6% F1, substantially outperforming Laya.

Confidence and selective prediction: - Jev confidence is useful for selective review: high-confidence subsets are consistently more accurate than unfiltered predictions, retaining ~48.7% of cases at 84.5% accuracy across PluRule. - Laya confidence is underconfident by 7.3–20.7 points and remains weak for automation decisions, even when its raw discrimination is competitive.

Connections

• Related to Content moderation via direct application to moderation systems and policy enforcement.
• Methodologically connected to Retrieval-Augmented Generation and in-context learning frameworks.
• Complements Harmful content detection literature by benchmarking a new model class (SOMs) against traditional classifiers like Detoxify.
• Addresses Policy violation detection — a distinct task from generic harmful-content classification.

Notes

Strengths: The controlled framework for isolating decision-context effects is rigorous. Cross-dataset evaluation and inclusion of both open-source and proprietary models is valuable. The distinction between policy-grounded and generic harmful-content tasks is important for real-world deployment.

Weaknesses: Only two SOM implementations evaluated (generalizability to broader SOM family unclear). Precedent retrieval relies on embedding similarity; semantic quality of retrieval not analyzed. No analysis of failure modes or misalignment between model confidence and actual correctness. Limited investigation into why Laya benefits less from retrieval.

Open questions: How do SOMs perform on multilingual or cross-policy settings? Can confidence calibration be improved post-hoc? Do different retrieval strategies (e.g., contrastive or semantic search) change precedent effectiveness?