Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature

Guangsheng Bao; Yanbin Zhao; Zhiyang Teng; Linyi Yang; Yue Zhang

In brief

LLMs like ChatGPT can fluently generate text, but detecting machine-generated content is critical for combating AI-enabled misinformation. This paper proposes Fast-DetectGPT, which uses conditional probability curvature — the observation that humans and machines exhibit different word-choice distributions in context — to detect machine text orders of magnitude faster than prior methods while improving accuracy by ~75%.

Summary figure for Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature

Contributions

Introduces conditional probability curvature as a foundational feature for distinguishing machine-generated from human-authored text, grounded in the observation that LLMs (trained on vast human text) develop higher average model probabilities than humans exhibit in individual word choices.
Proposes Fast-DetectGPT, a zero-shot detector that replaces DetectGPT's computationally expensive perturbation-based approach with efficient single-pass sampling, achieving 340× speedup.
Demonstrates superior detection accuracy across white-box (known source model) and black-box (unknown source model) settings on six datasets spanning multiple source models and domains.
Shows robustness to passage length variation, cross-domain transfer, and paraphrasing attacks.

Method

Fast-DetectGPT views text generation as sequential token selection. The key insight is that humans and machines select tokens differently given a context: LLMs, pre-trained on large corpora of human text, concentrate probability mass on fewer high-probability tokens, while individual humans spread probabilities more evenly. This manifests as higher average model probability for LLM outputs.

Rather than using raw probability (as DetectGPT does), Fast-DetectGPT measures conditional probability curvature — the shape of the conditional probability distribution at each token position. Specifically, for a passage $x$, the method:

Sample alternative word choices at each token position from the language model, capturing the conditional probability distribution.
Score by computing the conditional probability function: how much the log-probability of the passage deviates from the expected log-probability of random samples, normalized by the variance of those samples.
Compare the curvature to a threshold to classify human vs. machine.

The conditional probability curvature is formalized as: $$d(x, p_\theta, q_c) = \frac{\log p_\theta(x|\hat{x}) - \hat{\mu}}{\hat{\sigma}}$$

where $\hat{\mu}$ and $\hat{\sigma}$ are the expected score and variance of alternative samples. Unlike DetectGPT's approach (which requires ~100 perturbations per passage), Fast-DetectGPT evaluates samples in a single forward pass and scores all samples simultaneously via a unified scoring model, eliminating repeated model calls.

Results

Evaluated on six datasets (XSum for news, SQuAD for Wikipedia, WritingPrompts for story writing, WMT16 for multilingual text, PubMedQA for biomedical QA) across multiple source models (GPT-2, OPT, Neo-2.7, GPT-J, NeoX), Fast-DetectGPT achieves:

White-box setting: ~75% relative AUROC improvement over DetectGPT (e.g., 0.9967 vs. 0.9915 on ChatGPT generations).
Black-box setting: ~74.5% relative AUROC improvement over DetectGPT and competitive/better performance than supervised baselines.
Speedup: 340× faster than DetectGPT on Tesla A100 GPU (233 seconds vs. 79,113 seconds for 1.8B samples).
Robustness: Consistent detection accuracy across varying passage lengths and maintains high recall (87-98%) even under 10% false-positive tolerance.
Cross-domain: Outperforms DetectGPT on out-of-distribution datasets (PubMedQA) and generalizes across diverse source models.

Notes

The paper makes a novel conceptual contribution by framing machine-vs.-human text detection through conditional probability curvature, grounding it in the differing statistical properties of LLM outputs vs. individual human writing. The dramatic speedup (340×) without sacrificing accuracy is a substantial practical advance — DetectGPT's computational cost has been a barrier to deployment in real-world content moderation.

The work handles both white-box (access to the suspected source model) and black-box (unknown source) scenarios, with black-box performance particularly important for real deployments. Robustness experiments (passage length, cross-domain, paraphrasing attacks) are thorough.

One limitation: the method assumes pre-trained LLMs as the source of machine text. Its performance on text from small or highly specialized models is not explored. The ethical considerations section appropriately notes the risk of elevated false-positive rates on non-native English text (Liang et al., 2023) and recommends using varied and representative training corpora.