Skip to content
When Imperfect Automated Fact-Checking Backfires: Social Learning in the Presence of Misinformation

When Imperfect Automated Fact-Checking Backfires: Social Learning in the Presence of Misinformation

Authors: Zixuan Liu, Ming Cao
Venue: arXiv cs.CL, 2026
ArXiv: 2611.08053

TL;DR

Automated fact-checking tools are free and fast but error-prone; manual checking is costly but accurate. This paper studies whether adding a free automated tool alongside manual fact-checking improves social welfare (defined as the probability a cascade reaches the correct conclusion). Two effects collide: broader use of the automated tool helps agents escape wrong cascades, but its errors can mislead agents and initiate wrong cascades that manual checking would have prevented. For articles of intermediate reliability, if truthful articles are insufficiently informative and manual checking is costly, adding the automated tool strictly decreases welfare. On average across articles, the automated tool only improves welfare if its accuracy exceeds a threshold.

Contributions

  • Sequential social learning model for misinformation with two fact-checking methods: costly, perfectly accurate manual checking (MH) and free, error-prone automated checking (MA) with two-sided error rates.
  • Welfare characterization: thresholds on article reliability that determine when correct and wrong cascades arise, explicit in the cost and error rates of each method.
  • Formal proof that adding MA strictly lowers welfare on an intermediate reliability range when manual checking is sufficiently costly and truthful articles are not too informative.
  • Simulation evidence that MA can help on both sides of the intermediate range, and that platform integration of MA raises average welfare only above a threshold accuracy (≈ 0.81 balanced accuracy in their setup).

Method

Model setup: Agents arrive sequentially and decide to like or dislike a news article about which they are uncertain. Each agent observes: - The article's reliability r (prior probability it is truthful) - The actions of all predecessors - Optionally: a fact-check result (if they choose to check)

An agent's payoff is 1 if her action matches the article's true veracity, and 0 otherwise. Before acting, she may: 1. Pay cost K_H for manual checking, which is error-free. 2. Use the automated tool for free, which returns an error-prone verdict with type-I error e1 (false negatives: truthful articles marked fake) and type-II error e2 (false positives: fake articles marked truthful).

Agents are Bayesian and play a perfect Bayesian equilibrium. The dynamics are modeled as a Markov process on public likelihood ratios. A cascade occurs when all subsequent agents ignore their private information and take the same action.

Welfare definition: W(E, M) = P(correct cascade), the probability that the cascade (if one forms) matches the true veracity.

Main theoretical result (Proposition 2): When every agent who fact-checks prefers the automated tool (regime iii): - ∆W_T > 0 on one reliability interval (truthful articles benefit from broader checking) - ∆W_F < 0 on a different interval (fake articles harmed by automated errors) - If truthful articles are not informative enough (p < p̄), these two negative effects overlap on an intermediate reliability range, causing overall welfare decline ∆W < 0.

Simulation study: Monte Carlo simulations on a 50×50 grid of article reliability r and article informativeness p confirm the theoretical predictions and explore how welfare changes as the automated tool's accuracy improves.

Results

Theoretical findings: - When manual checking is sufficiently costly and automated tool is free, agents will use it in an intermediate range of posterior beliefs. - Adding MA strictly decreases welfare for articles of intermediate reliability (Proposition 2d) when p < p̄, where p̄ depends on the cost ratio K_H / (1 - K_H).

Simulation results (Fig. 3–5): - Welfare change is nonmonotonic in article reliability, with negative bands centered on intermediate-reliability articles. - As automated tool accuracy improves along the relation e1_A = 3e2_A, welfare first dips below manual-only baseline (regimes i–ii), then recovers only above a threshold balanced accuracy τ₀ ≈ 0.81. - Platform integration of MA below τ₀ reduces average welfare despite every user who adopts it gaining individually — a classic informational externality.

Connections

Notes

Strengths: - Novel and important question: how do error rates in automated tools propagate through social learning? Rarely studied jointly before. - Clean separation of two idealized methods (perfectly accurate but costly vs. free but error-prone) isolates the welfare effect cleanly. - Both theoretical (Propositions 2–3) and simulation results; theoretical results are sharp and interpretable. - Policy insight: platforms should not integrate automated fact-checking tools below a specified accuracy threshold, despite user-level adoption benefits.

Limitations & open questions: - Theoretical results apply only when manual fact-checking is error-free. Small errors in manual checking are not modeled, though the authors note simulations suggest robustness. - Welfare defined as "correct cascade probability" — leaves unmodeled the possibility of no cascade (agents still fact-check forever with vanishing probability). Discussion notes this is a feature of their assumptions (incomplete learning when neither method is free + perfectly accurate). - Does not model strategic behavior (e.g., agents might learn that automated tool is error-prone and discount its signal). - Assumes agents observe only predecessors' actions, not whether they fact-checked. Richer information structures might change incentives. - Open questions remain on analytical characterization of the accuracy threshold τ₀, optimal error-rate tradeoffs, and extension to >2 methods.

Relevance to the field: This addresses a timely design challenge: as AI-powered fact-checking becomes embedded in platforms, understanding when it hurts collective decision-making is crucial. The paper formalizes the intuition that cheap, imperfect tools can mislead not just individuals but cascades. The threshold finding (τ₀ ≈ 0.81) suggests a concrete platform requirement.