Topics covered
Motivation for watermarking
Why we need to identify LLM-generated text:
- Academic cheating: Large fraction of homework and term papers submitted with ChatGPT assistance
- Election misinformation and propaganda: Detecting state-generated social media campaigns
- Training data pollution: Excluding LLM output from future training data to avoid degrading model quality
- General attribution: Proving output came from a specific LLM, not just any model
The watermarking scheme
Core principle: LLM outputs are inherently probabilistic; we can replace some entropy with pseudorandom signals that encode a watermark without degrading output quality.
Method:
- At each token position t, the LLM generates probability distribution over possible next tokens
- Instead of sampling normally, use a pseudorandom function based on the previous k tokens (k-gram) to deterministically select tokens
- Selection rule: Choose token i that maximizes r_t,i^(1/p_t,i), where r_t,i is pseudorandom output
- This rule preserves the original probability distribution: if r_t,i were truly random, each token i would be chosen with probability exactly p_t,i
Detection:
- Given a document, compute all k-grams
- From k-grams, derive the pseudorandom values r_t,i
- Sum a function of these values across the document
- Watermarked documents yield systematically larger sums than non-watermarked ones
- Statistics determine confidence level with minimal false positives
Key properties:
- No quality degradation (output is statistically indistinguishable from normal LLM output)
- Negligible computational overhead at inference time
- Robustness to local modifications: survives word changes, reordering, deletion as long as k-grams are preserved
- Feasibility: requires O(1/α² × log(1/ε)) tokens, where α is average entropy per token and ε is acceptable error rate
- Practical: few hundred tokens for reasonable confidence, few thousand for >99.9% confidence
Attacks and limitations
Simple attacks that destroy watermarks:
- Translation: Translate to different language (e.g., French) then back → completely changes k-grams
- Pineapple attack: Insert specified word between each token, then remove → changes all k-gram structure
- Paraphrasing: Ask LLM to rephrase → new k-grams, destroyed watermark
- Summarization: Major edits remove signal
Theoretical limitation (Barak et al.):
An impossibility theorem shows that if an attacker can perform a rapid-mixing random walk over documents while preserving quality, any watermark can be removed. However, this bakes the conclusion into the assumptions—empirically, modifying documents may not actually be that easy.
Fundamental constraint:
Watermarking only works when there is entropy in token generation. Deterministic outputs (e.g., listing the first 1,000 prime numbers) cannot be watermarked.
Why OpenAI hasn't deployed
Competitive risk:
Some users dislike watermarking. Surveys show users might switch to competing LLMs that don't watermark (Anthropic, others).
Access control dilemma:
- Public detection tool: everyone can check for watermarks, but adversaries can also use it to iteratively modify text until watermark disappears
- Restricted access (universities, journalists, verified entities): requires expensive gatekeeping infrastructure; never built at OpenAI
- Private access: minimizes evasion but limits legitimate detection
Fairness concerns:
- ESL speakers using GPT to improve English fluency would have their text watermarked. Is that fair?
- Hard to design a watermark that works only on "unsympathetic" use cases
Organizational barriers:
- Complexity theorist (Aaronson) proposing idea part-time; pushing to product requires different skills
- No dedicated engineering team built the gatekeeping infrastructure
Current deployment status
Google: Deployed "SynthID" watermarking for some generative AI outputs (as of 2024), but detection tool requires application and approval—not publicly available.
Anthropic and OpenAI: Have not deployed watermarking at scale.
Regulatory and future directions
Legislation:
- California SB 942 (signed 2024): mandates watermarking for audio/visual content starting 2026, but text was removed from scope
- EU considering watermarking mandates
- Bills use vague language ("best currently existing technology"), unclear how enforcement would work
Research directions:
- Semantic watermarking: Embed signals at the level of ideas/meaning, not just token sequences
- Would survive paraphrasing and translation
- Requires modifying internal model representations
- Still unsolved; marked as open problem
- Clarify which classes of attacks can and can't be defended against under formal attack models
Coordination:
Aaronson advocates for formal coordination among OpenAI, Anthropic, and DeepMind to avoid tragedy-of-the-commons (each prefers no watermarking, but all would benefit from industry-wide adoption).
Notes
This talk updates the 2023 presentation with four years of follow-up: technique is sound and unchanged, but deployment remains blocked by business incentives and coordination failures rather than technical barriers. Aaronson is candid about the gap between "this is mathematically feasible" and "this ships as a product." The fairness questions around ESL learners and legitimate private use are the hardest to resolve.
Key takeaway: robust watermarking exists and could reduce academic cheating and state-level misinformation campaigns, but widespread adoption requires either regulatory mandate (EU, California SB 942) or industry-wide coordination that hasn't materialized. The impossibility result (Barak) is overstated—it assumes attackers can do things (rapid-mixing document rewrites) that may not be practically feasible.