
Contributions
- NoteHabitat benchmark: A 36-month linked archive of 102,994 annotation-related Weibo posts with 42,853 annotations and 431,407 comments, spanning content quality, engagement, reader response, and platform policy changes.
- Seven-task evaluation framework: Tasks operationalize correction context understanding across three dimensions:
- Content: annotation type classification, quality assessment (five dimensions: relevance, evidence support, objectivity, helpfulness, fluency), engagement prediction
- Interaction: comment stance detection, toxicity detection
- Management: policy-associated outcome prediction, temporal dynamics forecasting
- Systematic bias analysis: Models underrate annotation quality relative to human judgments, especially for evidence support and objectivity. Engagement predictions fail to capture the weak correlation between quality and visibility (ρ = 0.068).
- Platform-grounded evaluation: Unlike prior fact-checking benchmarks focused on annotation helpfulness alone, NoteHabitat captures the full social context: visibility constraints, contributor dynamics, audience response diversity, and platform policy effects over time.
Method
The benchmark captures annotation-related activity from Weibo Annotations, a crowdsourced fact-checking feature. The authors define correction context understanding as the ability to assess corrective content, interpret audience responses, and reason about platform dynamics within a shared fact-checking environment.
Seven tasks (T1–T7) operationalize this across a 3×3 matrix:
Content (T1–T3):
- T1: Annotate contribution type (information supplement, accusation/report, party response, non-annotation)
- T2: Rate annotation quality on five dimensions on a 1–5 scale
- T3: Predict engagement from annotation and source post text
Interaction (T4–T5):
- T4: Detect reader comment stance (support, oppose, question, other)
- T5: Identify toxicity in reader comments using the same contextual inputs
Management (T6–T7):
- T6: Predict policy-associated outcomes (submissions/contributions increase/decrease) following platform policy changes (P1–P5: eligibility expansion, feedback additions, rule documentation, revenue incentives, enterprise responses)
- T7: Forecast daily submissions, displayed annotations, and engagement trajectories using ABM (Agent-Based Model) simulation calibrated to observed data
Models evaluated: XGBoost, Random Forest, RoBERTa, DeBERTa, Qwen3-4B, Qwen3-4B LoRA, Qwen3.6, GPT-6-Astra, Gemini-3.1, DeepSeek-V4.1.
Results
Content and annotation quality (T1–T3):
- LLMs underestimate quality across all five dimensions. Fine-tuned RoBERTa and DeBERTa achieve the lowest zero-shot F1 scores on quality assessment, but still overestimate objectivity and helpfulness.
- Engagement predictions show weak correlation with quality (Pearson r = 0.068 vs. 0.085 for Mean baseline). Qwen3.6 achieves 0.085 r but with substantial prediction errors.
- Annotation quality and engagement are nearly uncorrelated in observed data; LLMs also fail to predict this correctly.
Interaction: Stance and toxicity (T4–T5):
- Aggregate accuracy for non-toxic comments reaches 97.93% (T4 baseline), but only 0.010 Toxic-F1 — systematic failure on the toxic minority.
- Supportive toxicity is a persistent blind spot: LLMs recall only 42.95% of supportive hostile comments vs. 60.85% of opposing comments.
- Fine-tuned LLM encoders recall only 53.55% of supportive hostile cases. Toxic-comment patterns reveal endorsement of corrections coexisting with abuse.
Management: Policy outcomes (T6–T7):
- GPT-6-Astra achieves 0.667 accuracy and 0.622 Macro-F1 on directional accuracy (T6) compared with 0.390 for Qwen3.6 (zero-shot).
- LLM-guided ABM forecasts correlate weakly (mean r = −0.101 to 0.163) with observed daily activity across model–outcome pairs. Policy-associated activity patterns are difficult to capture: a stable 20-day lookback helps some models but hurts others.
- Temporal fidelity (T7): GPT-6-Astra achieves 0.667 short-term and 0.622 long-term directional accuracy. Frozen LLM-guided ABM forecasts struggle with long-term temporal patterns.
Notes
The paper makes a methodologically strong argument: fact-checking models must understand not just individual annotation quality but the full ecosystem around corrections — visibility, audience reception, policy shifts, and temporal patterns. The weak quality–engagement correlation (ρ = 0.068) is a striking finding that challenges the assumption that "better" annotations automatically receive more attention. LLMs' systematic underestimation of evidence support and objectivity, especially in a non-English platform (Weibo), suggests that these models may struggle with grounded fact-checking in sociolinguistically diverse contexts. The toxicity blind spot on supportive hostile comments is particularly concerning for real-world deployment — abusive corrections can normalize misinformation while appearing collaborative. The temporal dynamics results (T7) underscore that platform effects unfold over weeks and months, not individual posts, and frozen LLM-guided simulations capture only static relationships. This is the first benchmark to jointly evaluate content, interaction, and management layers in a shared fact-checking environment, filling a gap between annotation-centric and platform-centric evaluation.