Watermarking LLMs to Fight Plagiarism

Tom Goldstein

In brief

Tom Goldstein describes watermarking techniques for detecting AI-generated text in language models. The method embeds hidden signals into generated text by partitioning vocabulary into "green" and "red" lists and incentivizing the model to favor green tokens; detectors can identify watermarked text from as little as 24 words with negligible false-positive rates. The talk also covers adversarial examples research and accidental data leakage from diffusion models like Stable Diffusion.

Topics covered

Watermarking for AI-generated text detection

The core contribution is a lightweight detection method that works by:

Dynamically partitioning the model's vocabulary into green and red word lists using a seeded random number generator
Giving the model a small reward to prefer green-list words during inference
Detecting watermarked text by counting green-list word frequencies—humans expect 50/50 splits; AI-generated text shows biased toward green
Requiring only dozens of tokens for high-confidence detection (false-positive rate ~10^-14 for 24 words)
Remaining accessible even with proprietary models (no white-box access needed for detection)

Robustness and attacks

The watermark resists several classes of attacks:

Summarization attacks fail because state-of-the-art summarizers like GPT-3 DaVinci preserve sentence fragments from originals, maintaining watermark signal
Paraphrasing attacks face challenges: if humans reuse bigrams or word fragments from the original, watermark signal survives
"Emoji attacks" (inserting emojis then removing them) can disrupt token-seeding, but require active prompt engineering (can be filtered server-side)
Trivial removal is always possible (replace text with random characters), but the question is whether quality-preserving removal exists; open-source models lack power for high-quality paraphrasing at scale

Incentive structures for adoption

Companies might adopt watermarking because:

Social responsibility: Watermarks help detect malicious uses (spam bots, social engineering, disinformation campaigns)
Data filtering: Watermarks help training datasets exclude previously generated AI text (following precedent from OpenAI's Whisper model)
Monetization: Companies can sell access to watermark detectors as a service

Adversarial examples research

Goldstein discusses his prior work on adversarial patches—physical patterns that fool object detectors (e.g., invisibility cloaks). Key findings:

Adversarial patterns transfer poorly across detector architectures and versions
Patterns trained on COCO dataset transfer well across person-detection architectures, but reliability is low
His lab stopped selling adversarial clothes commercially to avoid misleading users about their reliability

Data leakage in diffusion models

Goldstein's team discovered that Stable Diffusion accidentally reproduces training data:

Partial copying occurs in ~2% of generated images (when prompted with captions from training data)
The model replicates objects or backgrounds pixel-for-pixel from training images, though not entire images
This occurs at higher rates than other diffusion models tested (e.g., ImageNet-trained models)
Text conditioning (via prompts) may explain higher replication rates vs. class-label conditioning
Scaling detection to 2 billion training images poses infrastructure challenges (DNS server overload from queries)

Notes

This talk covers both a novel detection method (watermarking) and practical deployment considerations. The discussion of watermark robustness is more nuanced than online critiques suggest—while watermarks are removable in principle, removal via paraphrasing currently requires quality trade-offs with available open-source models. The diffusion model data-leakage findings highlight that accidental reproduction, while quantitatively rare, is still a concern for large-scale generative models trained on web data.