Skip to content
The uneven rise of AI-generated web content

The uneven rise of AI-generated web content

Authors: Adam Visokay, Sounyakanti Pant, Molly Griston, Polina Isakova, Mark Nepf, Denis Peskoff, Riya Sinha, Jikai Sun, Isabelle Thiang, Letao Wang, Yiwel Wu, Lauren Yan, Tyler H. McCormick

Venue: arXiv cs.CY, October 2026 — arxiv:2610.36631

TL;DR

Analyzing 5+ million webpages from 2019–2026, this work documents that AI-generated text prevalence rose from ~2% to nearly 25% by August 2026, with uneven adoption across the web. Growth is concentrated in peripheral, poorly-connected domains rather than central platforms; English-language and .com pages show higher prevalence than .edu and .gov sites. The authors use multiple AI detection tools with a Bayesian hierarchical model to correct for detector error, showing that growth patterns persist after adjustment.

Contributions

  • Large-scale longitudinal dataset: 5.1 million crawlable webpages spanning 2019–2026, 71 Common Crawl snapshots, stratified by PageRank centrality.
  • Network-stratified analysis: Demonstrates that AI-generated text growth is concentrated on peripheral (low-PageRank) domains (70.5% of eligible pages, 78.8% of net increase), while central tier domains show slower adoption.
  • Multi-detector error correction: Employs five AI detection models (Open Pangram, Llama2-Classifier, Binoculars, Falcon-7B, commercial Pangram 4) and uses a Bayesian hierarchical model to adjust for language-specific and context-specific false positive and false negative rates.
  • Language and domain-level stratification: Reveals that English-language pages (32.8% AI prevalence, Aug 2026) exceed non-English pages (17.0%), and that .com domains (26.7% prevalence) substantially exceed .edu (11.2%) and .gov (11.6%) domains.
  • Growth rate acceleration: Documents that annual growth in AI-generated text has accelerated—0.2 percentage points per year (Jan 2019–Feb 2023), 4.6 pp/year (Feb 2023–Jan 2025), 8.1 pp/year (Jan–Aug 2026)—with >3× acceleration in recent months.

Method

The work uses a two-stage stratified cluster sampling design applied to Common Crawl's full hyperlink graph (71 snapshots from Jan 2019 to Aug 2026):

  1. Sampling design: PageRank tiers are computed for each crawl; domains are stratified by six PageRank cutoffs defining tiers (Top, Upper-middle, Middle, Lower-middle, Bottom, Tail). Within each tier, pages are sampled proportionally, excluding boilerplate content (pages <500 characters, navigation, login).

  2. AI detection: Five detectors are evaluated on labeled benchmarks (Pangram, Llama2-25B-Chat, Binoculars, Falcon-7B, Pangram-4). Detectors' output is treated as noisy classification of a binary latent variable (page is / is not AI-generated). Sensitivity and specificity are estimated from benchmark data and allowed to vary by language, domain, and text length.

  3. Bayesian hierarchical model: Posterior prevalence estimates combine detector outputs with language-specific error rates and pre-2022 control crawls (low-prevalence baseline). Poststratification uses page-count weights to aggregate cell-level estimates to population totals, accounting for heterogeneous page counts across tiers, languages, and domains.

  4. Uncertainty quantification: Credible intervals propagate detector uncertainty into prevalence and growth estimates. Sensitivity is examined across different detector subsets and error-rate assumptions.

Results

  • Overall prevalence trajectory: Estimated share of pages with fully AI-generated or AI-edited text rose from ~2% (Jan 2019) to ~24% (Aug 2026, 95% CI: 22.9–24.4%).
  • Growth by network centrality: Bottom and Tail tiers (70.5% of eligible pages) account for 78.8% of the net increase; central tier (Top, Upper-middle, Middle) shows prevalence rising from ~13% to ~27%, while peripheral tiers rise from ~0.8% (Dec 2021) to 26.5–27.5% (Aug 2026).
  • Language disparity: English-only pages: 32.8% (32.2–33.4%), non-English-only pages: 17.0% (15.4–18.4%), bilingual: 20.2% (19.3–21.0%). English-only prevalence grew 10.83 pp/year in second interval (Jan 2025–Aug 2026) vs. 4.82 pp/year for non-English-only.
  • Top-level domain (TLD) variation: .com: 26.7%, .edu: 11.2%, .gov: 11.6%, .net: 15%, .org: ~14%, other: ~14%. After standardizing for page composition (TLD, language, text length, centrality tier), differences persist and exceed 1 percentage point.
  • Growth rates stabilizing: Acceleration from 0.2 pp/year → 4.6 pp/year → 8.1 pp/year suggests continued rapid increase, though the slowing of recent acceleration (8.1 pp/year vs. implied >10 pp/year if early-2026 slope continued) may indicate approach to saturation in some domains.

Connections

Notes

Strengths: The paper's key contribution is disambiguating where AI-generated text is growing (network periphery, not center) and controlling for detector error. The Bayesian hierarchical model is transparent and allows uncertainty from detector performance to flow into prevalence estimates. The multi-detector approach, language stratification, and explicit accounting for compositional effects are methodologically sound. The dataset and design (full web graph stratification) is novel and scale of 5M pages over 7.5 years is substantial.

Weaknesses and open questions: - Detector choice: The paper commits to five detectors but all are general-purpose classifiers; specialized detectors (e.g., domain-specific fine-tuning) might change prevalence estimates. The authors acknowledge detector bias (e.g., Open Pangram overestimates on .com pages) but do not investigate whether estimates are robust to held-out detectors. - "AI-generated or AI-edited" conflation: The operational definition includes edited text, which conflates minor touchups with wholesale generation. Prevalence of "fully generated" vs. "human-written with AI revision" is not separately quantified, limiting utility for some policy questions. - Temporal alignment: Detector performance may itself drift over time (old detectors fail on newer models); the paper uses fixed detectors across all snapshots. Re-training on time-stratified benchmarks would strengthen causal claims about prevalence growth. - External validity: Common Crawl samples primarily English and crawl-friendly content; pages behind authentication, paywalls, or CAPTCHAs are excluded. The results likely overstate English-language prevalence and understate prevalence on secure/private content. - Causality: The paper documents correlation between network centrality and AI prevalence but does not isolate mechanisms. Is adoption slower on central sites because of editorial policy, technical/policy constraints, audience expectations, or other factors?

Why it matters: The finding that AI-generated text is more prevalent on the periphery than the center has implications for information integrity and language model training. If training corpora increasingly consist of peripheral content (blogs, small forums, auto-generated pages) rather than high-quality central sources, future LLMs may degrade as they absorb more machine-generated training data—a potential feedback loop. The language and TLD disparities also suggest that regulation, institutional norms, and platform policies do influence AI content adoption, though mechanisms are not yet clear.