Harm Is Behavioural, Moderation Is Lexical

Authors: Gabriel Lars Sabadin, Nicola Harding, Sarah Kingston

Venue: arXiv:2610.31047 (cs.CL) — arXiv

TL;DR

Automated content moderation systems recognize harm based on lexical form rather than behavioral function, creating a double inequity: dialects and minority languages face over-flagging while genuine harms (like manipulation tactics) go undetected in under-resourced languages. The paper unifies fairness literature (false positives in African American English, drag-queen discourse) and robustness literature (under-protection in low-resource languages) as consequences of the same root cause, proposes an evaluation protocol that reports the gap between best- and worst-served language communities as the headline metric, and demonstrates the method on five systems across eleven languages.

Contributions

(1) Unifies two problem streams: Connects over-flagging of dialects/identities (fairness) and under-protection in low-resource languages (robustness) as evidence of a single cause: systems trained on lexical form rather than behavioral intent.

(2) Distinguishes behavior from vocabulary: Argues that harmful behaviors (grooming, manipulation, threats) are stable cross-linguistically; harmful vocabulary is not. Lexical systems miss behavioral harms in languages where the harmful vocabulary differs.

(3) Taxonomy of seven failure modes: Over-flagging (dialect penalty, in-group language framed as attack, identity terms as false signals, annotator perspective bias), under-protection (low-resource collapse, mixed-language code-switching, behavior without vocabulary). Each mode produces a distinct error pattern.

(4) Proposes evaluation protocol: Recommends measuring gap between best- and worst-served language varieties as headline result (not per-variety accuracy tables). Includes contrastive pairs to isolate function from form, permutation p-values to correct for small sample sizes, and per-language error reporting in both directions.

(5) Demonstration on public systems and test suites: Tests five open systems (keyword lexicon, HurtLex, QwenJGuard-0.6B, Qwen3Guard-8B, PolyGuard) on eleven languages using HateCheck and Multilingual HateCheck. Finds one system where all five systems fall short: no system meets native-speaker accuracy standard in any language on implicit-hate detection.

Method

Problem formulation: The paper argues that systems define harm lexically (by keywords and surface features) rather than functionally (by what the message does). This creates two observed failure modes:

• Over-flagging: A lexical system that associates African American English features with slurs will flag the same words in contexts of solidarity or reclamation. Markers like ***** or profanity used for emphasis, group affection, or minor rule-breaking become misread as abuse.
• Under-protection: A system trained only on English hate speech won't recognize the same behavioral harm expressed through unmarked vocabulary (a polite manipulation tactic, a secretive threat) in a low-resource language.

Hypotheses: - H1 (discrimination): Systems trained on lexical form will rank harmful cases above harmless look-alikes across languages, but will fail reliably on contrast cases (same function, different form). - H2 (stability): A system recognizing function will show smaller error gaps across languages; one recognizing form will show large gaps where form diverges from function.

Evaluation protocol (Section 6):

1. Native-authored parallel test sets: Each harmful and harmless case written or validated by native speakers of each language, not machine-translated.
2. Contrastive pairs by function: Each harmful case has a harmless counterpart that shares as much surface form as possible (e.g., reclaimed slur vs. slur in attack; benign profanity vs. hostile profanity). Isolates function from form.
3. Error reporting both directions: For every language and dialect, report false-positive rate on harmless cases and false-negative (miss) rate on harmful ones, with confidence intervals. Neither alone is meaningful; the ranking of which language is under-served requires both.
4. Gap as headline: Report best-minus-worst language gap at a fixed operating point, with permutation p-value. Allows regulators and platforms to see at a glance whether a system is biased.

Systems tested: - Keyword lexicon (HurtLex): Static word lists, over 50 languages - Toxicity classifier (TextDetox, v2): Large RoBERTa model, 14 languages - QwenJGuard-0.6B: Small LLM guard, 119 languages - Qwen3Guard-8B: Larger LLM guard, same family - PolyGuard: Multilingual moderation model, 17 languages

Test suites: - HateCheck (Röttger et al., 2021): English cases, written/validated by experts - Multilingual HateCheck (Röttger et al., 2022): 10 additional languages (Arabic, Dutch, French, German, Hindi, Italian, Mandarin, Polish, Portuguese, Spanish) - Implicit-hate cases (implicit-derogation templates: no slurs, no profanity, just manipulation, exclusion, coded language)

Results

Headline findings:

1. Lexicon coverage drives gaps: The keyword lexicon's coverage is 52.7 percentage points wider for English (52.7% harmless flagged) than Hindi (0.0% — no coverage), and its gap on missed implicit hate is 45.7 points (English 19.7% missed vs Arabic 65.4% missed). The gap grows with capability: larger models narrow absolute errors but widen relative gaps.

2. Same language, both failures: PolyGuard exhibits both over-flagging (69% of English harmless contrast cases, e.g., reclaimed slurs and benign profanity) and under-protection (77% of implicit hate missed in Hindi). The errors are coupled: form-based flagging catches reclaimed-slur pairs but misses behavior.

3. Discrimination without calibration: Across all five systems, 51 of 55 language-system combinations fall short of the 0.7 balanced-accuracy standard (native-speaker annotator performance). The systems show calibration only for English and French (narrow confidence intervals, good AUC); for Arabic, Hindi, Mandarin, gaps remain large even when aggregate accuracy looks acceptable.

4. Implicit hate exposed: The small QwenJGuard guard model shows no function recognition (coupling = −0.23, p > 0.05 — its errors don't correlate across languages). It over-flags 40–50% of harmless cases uniformly, and under-detects implicit hate 54–77% of the time across languages.

5. Best-worst gap at fixed operating point: At the point where the lexicon achieves balanced accuracy on English, its gap between English and Hindi on implicit hate is 57.1 percentage points (English 22.7% expected under-service vs Hindi 80% expected). For the 8B guard model, the gap is 68.6 percentage points (English 23.5% vs Hindi without Hindi support, assumed 92%).

Connections

• Related to work on racial bias in hate speech detection (Sap et al., 2019) via shared finding that surface features (e.g., African American English markers) create systematic bias in annotator and model behavior.
• Related to work on drag-queen language misclassification and context-blindness in moderation (Dias Olivia et al., 2021).
• Related to work on representation disparity in NLP across 7000+ languages (Joshi et al., 2020).
• Related to work on multilingual LLMs and English-centric safety training (Deng et al., 2024).
• Related to HateCheck test suite as methodological foundation for contrastive evaluation (Röttger et al., 2021).

Notes

Strengths: - Reframes a false dichotomy (fairness vs robustness) as a single phenomenon, focusing attention on the root cause (lexical vs functional definition of harm). - Clear taxonomy of failure modes with concrete examples and published evidence for each. - Evaluation protocol is practical and readily applicable to commercial systems; gap-as-headline metric directly actionable for regulators. - Diverse systems tested, including open models and academic test suites; no proprietary APIs; full reproducibility reported. - Addresses behavioral harms (grooming, manipulation, threats) that are genuinely hard to encode lexically and persist across languages.

Limitations: - Tested only on hate speech and implicit-hate templates; does not address scams, coercion, or other functional categories. - Test suites are small (20–145 cases per language); confidence intervals reflect this; permutation p-values sometimes inconclusive. - Code-switched and mixed-script cases are synthetic templates, not representative of how users actually code-switch in practice. - Does not measure system retraining dynamics or how labels change over time (words that perform harm change faster than functions). - Paper evaluates recognizing function, not defining it; the definition of what constitutes harm (whose harm counts, harm to whom) remains a policy choice, not a technical one.

Research directions: - Extend protocol to scams, coercion, sexual exploitation, and other behaviorally-driven harms; investigate whether form-function gap persists across harm categories. - Fund native-speaker data collection and annotation for low-resource languages to close the under-protection gap. - Study how systems trained on behavior-annotated data behave; compare to current lexical/form-trained systems. - Investigate whether architecture choices (LLM vs. classifier vs. lexicon) predict success on function-based evaluation.