Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text

Abhimanyu Hans; Avi Schwarzschild; Valereia Cherepanova; Hamid Kazemi; Aniruddha Saha; Micah Goldblum; Jonas Geiping; Tom Goldstein

In brief

Binoculars is a zero-shot detector for LLM-generated text that requires no training data or model-specific modifications. It views text through two language models and computes a score based on the ratio of perplexity to cross-perplexity, achieving over 90% detection of ChatGPT-generated samples with 0.01% false positive rate and generalizing across multiple LLMs including GPT-3.5, LLaMA-2, and Falcon.

Summary figure for Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text

Contributions

Identification of cross-perplexity as a powerful zero-shot signal for detecting LLM-generated text, addressing the "capybara problem" where prompt-dependent perplexity fails.
Binoculars: a zero-shot detection method that combines perplexity measurements from two language models without requiring training data, model finetuning, or access to model internals.
Comprehensive evaluation showing state-of-the-art accuracy across multiple datasets (News, Creative Writing, Student Essays), document sizes, languages, and LLM architectures.
Robustness analysis demonstrating reliable detection despite modified prompting strategies, memorized text, and other adversarial variations.

Method

Binoculars operates through a two-lens approach that compares how two different language models score the same text:

Perplexity Measurement: For a given text, compute the log perplexity under an "observer" language model M₁: $$\log PPL_{M_1}(s) = -\frac{1}{L} \sum_{i=1}^{L} \log(Y_{i,i})$$ where L is the sequence length and Y represents the next-token probability distribution.
Cross-Perplexity Computation: Compute next-token predictions using a "performer" language model M₂ and compute cross-perplexity — how surprising M₁'s observed text is to M₂: $$\log \text{X-PPL}_{M_1, M_2}(s) = \frac{1}{L} \sum_{i=1}^{L} M_1(s_i) \cdot \log(M_2(s_i))$$
Binoculars Score: Normalize perplexity by cross-perplexity to obtain the detection score: $$B_{M_1, M_2}(s) = \frac{\log PPL_{M_1}(s)}{\log \text{X-PPL}_{M_1, M_2}(s)}$$

Key Insight: LLM-generated text exhibits a characteristic signal where the ratio of perplexity to cross-perplexity distinguishes it from human-written text, solving the capybara problem where prompts can manipulate raw perplexity scores.

Results

Zero-shot detection across domains: Binoculars detects 96–99% of ChatGPT-generated text at 0.01% FPR across News, Creative Writing, and Student Essays datasets, outperforming commercial detectors (GPTZero) and open-source baselines (DetectGPT, Ghostbuster, DNA-GPT).

Cross-model generalization: The detector works on LLaMA-2-13B, Falcon-7B, GPT-Neo, and other open-source models without model-specific modifications or training data.

Robustness to adversarial variations:

Handles modified system prompts (Carl Sagan, Non-Robotic, Pirate styles)
Detects memorized famous texts correctly as human-written
Maintains high accuracy on grammatically-corrected essays and non-native English writing
Achieves 92% detection accuracy on GPT-4 and 96.89% on Gemini-Pro-generated text

Document size adaptability: Performance remains strong even with short documents (128 tokens), unlike some baselines that require longer passages.

Multilingual capability: Tested on Urdu, Russian, Bulgarian, and Arabic with high precision (>80% in most cases).

Notes

Strengths: The zero-shot property is a major practical advantage — no need for training data, which is often expensive or unavailable. The method generalizes remarkably well across different LLM families and architectures. The evaluation is thorough, covering multiple document types, languages, adversarial variations, and recent commercial APIs. The identification of cross-perplexity as a signal is theoretically motivated and empirically compelling. The paper is clearly written with reproducible methodology.

Limitations: The method requires access to log probabilities from at least two language models, which limits applicability to closed-source APIs (though the paper evaluates GPT-4 and Gemini). Performance on very short texts (<128 tokens) is lower than on longer documents. The detector may be fooled by advanced adversarial prompting strategies designed specifically to evade detection, though the paper does not exhaustively test such strategies.

Open questions: Whether Binoculars remains effective as LLMs continue to improve and their outputs become indistinguishable from human text. How the detector performs against future models trained specifically to minimize detectability. Whether the method extends to other modalities (synthetic images, video) or requires adaptation.