I notice the "Limitations" section talks about how content only at some point touched by Claude may return a positive, and content that returns a negative may still be Claude generated. But I really would have liked for them to state explicitly that entirely false positives where a piece is fully human-written may still be marked as generated, because too many institutions with the power to ruin someone's life over that have trouble understanding the concept.
> When a supported Claude model generates text, it weaves an imperceptible watermark directly into the text itself. You won’t see it, and it doesn’t change the meaning, quality, or readability of Claude’s response.
I'd like to know a lot more about how that works.
A lot of my interactions with Claude return pretty precise text. If I ask it to edit a project and refactor a specific function in several places I know exactly what I want to happen, it will NOT be OK if those refactors have some kind of weird pattern baked into their text to act as a watermark.
I guess this may be covered by this:
> Content generated by Claude may not carry a detectable mark if, for example: [...] The passage is very short, leaving too little text for a reliable signal;
>I'd like to know a lot more about how that works.
My guess is that it works like Gemini's SynthID: by altering the logprobs of the next token.
Like, for every 10th token, instead of outputting the most probable, it outputs the 17th most probable, or something. (Obviously it's way more complicated but I think conceptually this is how it works.) No human will notice this, but a classifier trained on Claude's output will.
So it's not like the watermark is the words "le epic bacon" and Claude will output "le epic bacon" in everything. That would be extremely annoying (and easy to defeat).
They still need to choose when to do that though. When I prompt the program to e.g. alter a bash script in a specific way or to recite a longer known text it can't go round and randomly exchange tokens. It has to somehow define what is a simple repeated text from a different origin and what is a novel generation.
Though if pangram should be trusted, there are still statistical artifacts that tells you that a text LLM generated. I don't find that to be implausible.
What happens if someone handwrites a Claude output, then someone uses that handwritten text as a reference.
Now you've got a watermarked idea which may have no direct linkage to the usage of Claude.
Because there's an expectation of authenticity from the written word.
If you've referenced something handwritten, you don't expect it to be the output of an LLM.
Similarly, if you quote someone word-for-word, you wouldn't anticipate their words to be flagged as Claude content, but if someone memorized Claude output word-for-word. That would still be classified as a Claude output.
Going forward you could categorize the influence of Claude on a population based off a percentage match between their spoken words with the LLM prose.
Most likely watermark will be proportional to the input/output ratio, i.e. if you input a long document and ask to make edits, it will not attempt to watermark it. On the other hand, if you provide a tweet and ask it to write an article, that will include watermark. Just a guess (and yes, it feels flawed)
I can tell you how: Claude produces a huge wall of text with jargon ridden bullshit and invented terms no human subject matter expert would seriously use and overuse.
I've heard that this kind of watermarking process works by biassing the statistical sampling towards a partition of the set of possible next tokens (red set and green set), at each position. It might only be a slight nudge each time, but over a sequence of tokens, the likelihood of repeating the bias by chance is increasingly improbable.
The bias is different for each position and follows a defined RNG, seeded somehow predictably.
Can be either an open algorithm, or not. If not open, then an API could be provided to determine if text is watermarked or not.
How it applies to code - maybe it could be a subtle nudge to symbol names, etc, I'm just speculating (I only read about this in passing very recently).
There's a computerphile video (https://www.youtube.com/watch?v=XZJc1p6RE78) with Dr. Mark Pound explaining a paper by John Kirchenbauer, Jonas Geiping et al. (https://arxiv.org/abs/2301.10226) that described a method for watermarking LLM output like this. It's not directly stated anywhere in the Claude support article that this is what they're using, but the properties of the watermark described seem to point to this method.
Based on my understanding, it can only be applied to code in very limited ways: docstrings, variable names, string literals. The code itself can't really have tokens changed to another equally correct token (the foundation of the watermark) because then the code breaks! And the few places that you can do so are likely erased by formatters anyway.
So my code that Claude makes, which previously was using the best (most probable) tokens for the job, will now be getting worse in random positions, to appease a voluntary EU suggestion. Love that.
I have had a hunch for a while now that (in addition to these tools), Anthropic has actually leaned in to Claude's distinctive manner of writing since it makes the text more obviously AI generated and thus less susceptible to misuse.
That's not necessarily the same thing as a markov-style fingerprint but it could be a correlated factor.
It could also partly be a byproduct of examples of claude writing being in the dataset, which of course anthropic has lots and lots of and they do train on.
no way. there's just no good excuse for why "load-bearing" and "worth flagging" are everywhere now, I've pretty much never seen that in the wild before
I suspect it's because of alignment concerns. The more deeply they can integrate their principles, the harder it'll be to misuse. Or at least that's the idea.
It's pretty trivial to command it to not speak that way. That's one of the first things you should write into the prompt. What style you want it to write in. Make it use a very concise and dry academic style with no overt LLMisms, melodramatic or flowery language, or metacommentary.
Can someone help me understand how exactly this watermarking of text works?
Given that text is, well, text, and not some kind of binary format, I don't see how any watermarking can work unless you insert characters which are invalid under Unicode. I further don't really understand how this won't be perceivable by assistive technology (the "watermark" will just appear as either unreadable characters, or if the watermark is mixed thoroughly enough into the text, it will scramble the text to any speech synthesizer and will make it really really obvious). Thus, I don't see how this wouldn't be insanely trivial to remove. And this is before we get into things being put on the clipboard. Sure, I can press the "Copy" button at the end of each response, but what I can also do is manually select the response and copy it, or only copy partial selections, or any number of other things. How does this "watermark" (or any "watermark" technology) take into account this?
So, really, to summarize this: I see no way of this actually being technologically achievable unless we revise the very core of how computers work and encodings for textual information. So I'm very curious as to how this is actually supposed to work.
Have a look around for token biasing, or green lists. It's based on a nudge to the choice of the next token (which can always be drawn from a set of possibilities which are all probable enough).
At first I thought this approach was just the "LLM flavour" of writing, but it's way more subtle, especially as the bias is applied uniquely for each token position.
Yeah, will do, this sounds interesting since I'm not entirely sure how this would actually be reliable to any degree. Thanks for the help, not sure why I got downvoted since I was genuinely curious.
it's a statistical way. like for example maybe in your above paragraph claude maybe writes "Thus, I don't see how this wouldn't be insanely <easy>(instead of trivial) to remove" and then also says like "And this is before we <analyze> things being put on the clipboard." or maybe the i just says the word "the" in a certain pattern or frequency.
you can then consistently like figure out if it was claude that wrote the sentence. it is easy as you noted if you just get another ai to read it and then rewrite it.
If Claude was the only model family they could ship a change like this and users who want to cheat (or don't like watermarks for other reasons) would just have to put up with it.
In a world with many different competing models, the risk of losing customers to other providers over this is much more real.
Maybe they've looked at the numbers and the portion of people who clearly use Claude to cheat on examples etc is so tiny that losing them to other providers isn't a problem?
Scott Aaronson spoke about this in a colloquium where he said that this was mooted at OpenAI before the decision was made by Altman to not implement it for the reasons you describe.
I’m more worried that this will degrade performance. I want the best results from a model, not the results that fit a constraint that’s not defined by me. Any increased cost or latency is also unacceptable.
> We’re also working to enable users and other third parties to detect Claude’s embedded watermarks and provenance metadata.
This seems to be similar in execution to Google's SynthID. I hope they release actual code the technically proficient can use, unlike SynthID which can only (afaik) be queried with Gemini's UI.
Thankfully, there are a variety of Chinese models that never will. I think we all know that in a few years, they will also be the only relevant offerings on the market, due to not being bogged down with over-zealous ""safety"" footguns.
I am cancelling my Claude max 5x subscription and moving to ChatGPT pro. I have difficulty enough trying to ensure my meaning comes through correctly, along with everything else; to now have to look out for/analyse watermarks too?
I feel shamed enough by society, thanks Anthropic.
How exactly does this impact proofreading? You can manually apply the suggestions (typo here, unnatural sounding sentence there, etc.) the LLM gives you to your own content, and it would stay watermark-free.
Unless with "proofreading" you actually mean having the LLM write your content for you.
> Generated text will carry embedded watermarks, and generated files will include digitally signed provenance metadata where supported.
This should make it easier to catch cheaters who use Claude, right? Unless everyone runs their artifacts through some watermark and metadata sanitizer?
The approach Pangram has taken which works pretty well is to simply lower the recall a lot but ensure the precision is very high. Which means potentially high false negative rate but low false positive rate.
I don’t like the idea of hacking a response to contain a watermark. I also don’t like the idea of false positives detections coming directly from Anthropic. If people read more AI generated content, people will probably start writing more in that style
The flip side of this is that if AI-generated content becomes reliably identifiable and carries a stigma, then people might deliberately change their styles to be more diverse and human.
One example I've seen are junior employees at my company deliberately adopting a lowercase/less punctuation writing style so as to stand apart from AI.
If it's that simple and obvious, you'll have 10 "Remove Claude Watermark" web-apps by the end of Day 1. Most of them coded by Claude.
Hell, it'll probably happen no matter how sophisticated their watermark is. There's no watermark in text that can't be detected and removed, and no text that can't be converted to generic keyboard ASCII.
You forgot about the cases where (1) people don't care, (2) people want to say "I used an LLM for this". I'm convinced that those cases happen more often than you think. Why not cover them with a simple mechanism? It's also in the interest of AI companies who don't want to train on AI output.
Sure, but let's first find out how many % of people are willing to be frank about their AI usage, and/or don't care about it. My guess is it is worthwhile to do this.
Those invisible spaces get wiped by the first sanitizer in any normal ide. Worse it'll instantly break parsing for configs like yaml where spaces are critical for structure
Reactionary emotional advice does no good, it just makes people want to hold their positions more defensively. If you care enough to say something, care enough to say it with reasons that might shift someone’s perspective.
Does it mean their models will always write slop? Making the writing non-collapsed to specific patterns seems to break any injected/learned fingerprinting.
If the western AI companies are forced to comply with this type of BS, and develop their models to do their job while balancing a book on their head and hopping on one foot, the Chinese models just got a free pass to completely dominate the frontier.
What's the problem, really? Given the direction the U.S. has been heading in recent years, I wonder what really sets it apart from China. Europe needs to maintain an equal distance from both the U.S. and China.
So this won't be happening in the US, but in the EU:
"When a supported Claude model generates text, it weaves an imperceptible watermark directly into the text itself. You won’t see it, and it doesn’t change the meaning, quality, or readability of Claude’s response.
Because the watermark is part of the text, it will travel with the text when it’s copied and pasted elsewhere, and may persist through some editing. Watermarking will be applied at the model level, which means it will be present no matter which Claude product or surface the text comes from."
I would love to see what this looks like in practice. Especially in generated code. I assume this is more than insertion of non visible special unicode whitespace characters, but more in the pattern of the text content itself?
It essentially looks like the difference between two different runs of the model with the same prompt but different seeds. The watermark is essentially a small bias in the model such that when there's multiple different tokens that could conceivably follow the previous token, the model will only pick some subset of them (the subset is derived from a hash of the previous token). This bias can then be checked for statistically (without needing access to the model and without needing the whole prompt), and for longer text where there's enough freedom in word choice you can show that it would be vanishingly improbable to accidentally follow the rules in the watermark.
Models can't reliably follow instructions involving their own logprobs unless they can take agentic control and use quite sophisticated dynamic grammars/structures/constraints to force this behavior in one shot (which can be slow and the dynamic grammar modification feature isn't supported in closed model APIs for safety reasons) or repeated attempts at rewriting which is expensive/slow.
Yes they can do this, but it's more likely closer to the original "red token, green token" paper: https://arxiv.org/abs/2301.10226
i.e. take half of your LLMs vocabulary, and upweight its probabilities by ~55% to the other half's ~45%, and scan for overuse of this half of all tokens. You can even choose a different half/slice for every individual user, for every individual action. You can implement this under the hood cheaply with logit-biasing.
IIRC, watermarking text could be as simple as training the model to use specific words/phrases more frequently than what you would expect to find in human-written text, to the point where it's highly statistically improbable that it wasn't AI generated. I assume similar logic could apply to code in the form of functions/code styling.
That's probably an over simplification. Also a solid defence that can be used against complaints about the way AI writes text.
No, they are the ones making claims, especially their CEO saying things like a 1/10000 false positive rate. Their own testing showed a 2% rate, which is insanely high when you talk about the number of papers students turn in. Worse their testing methodology compared it with pre-llm documents and not post llm documents that were human written (much harder and more expensive to verify), by treating language as static.
Pangram is subjectively very useful and I personally subscribe, but the burden of proof is on them. The product is very much "trust me bro" and I fear that if they ever try to improve recall both their precision and reputation will tank.
I notice the "Limitations" section talks about how content only at some point touched by Claude may return a positive, and content that returns a negative may still be Claude generated. But I really would have liked for them to state explicitly that entirely false positives where a piece is fully human-written may still be marked as generated, because too many institutions with the power to ruin someone's life over that have trouble understanding the concept.
> When a supported Claude model generates text, it weaves an imperceptible watermark directly into the text itself. You won’t see it, and it doesn’t change the meaning, quality, or readability of Claude’s response.
I'd like to know a lot more about how that works.
A lot of my interactions with Claude return pretty precise text. If I ask it to edit a project and refactor a specific function in several places I know exactly what I want to happen, it will NOT be OK if those refactors have some kind of weird pattern baked into their text to act as a watermark.
I guess this may be covered by this:
> Content generated by Claude may not carry a detectable mark if, for example: [...] The passage is very short, leaving too little text for a reliable signal;
>I'd like to know a lot more about how that works.
My guess is that it works like Gemini's SynthID: by altering the logprobs of the next token.
Like, for every 10th token, instead of outputting the most probable, it outputs the 17th most probable, or something. (Obviously it's way more complicated but I think conceptually this is how it works.) No human will notice this, but a classifier trained on Claude's output will.
So it's not like the watermark is the words "le epic bacon" and Claude will output "le epic bacon" in everything. That would be extremely annoying (and easy to defeat).
They still need to choose when to do that though. When I prompt the program to e.g. alter a bash script in a specific way or to recite a longer known text it can't go round and randomly exchange tokens. It has to somehow define what is a simple repeated text from a different origin and what is a novel generation.
I am wondering how that applies to newly generated code.
Odd variable naming? Stylistic choices that are watermarked?
Or as someone else noted further down in the comments, it could be more subtle:
Between the first and second most likely choice, in certain positions it will consistently choose in a certain way.
> Odd variable naming? Stylistic choices that are watermarked?
Whatever it is, I'm sure it's load-bearing.
You're absolutely right. But it is not just load-bearing, it is the load-bearing seams.
It was quick :) … https://claudewatermarkremover.app/
Though if pangram should be trusted, there are still statistical artifacts that tells you that a text LLM generated. I don't find that to be implausible.
Most likely this method https://arxiv.org/pdf/2301.10226 (EDIT: and Google's SynthID paper which builds on it https://www.nature.com/articles/s41586-024-08025-4)
That "just add a constant to the green logits" as a fix to the entropy problem is so elegant I love it.
> I'd like to know a lot more about how that works.
Count load-bearing words using two different algorithms in a belt-and-braces fashion
One thing worth flagging: those words are load-bearing
Fair - I should have been honest about the watermark.
You’re absolutely right. Yo momma is doing a lot of heavy lifting here. Her load-bearing methods have the right shape.
Belt, braces, and suspenders.
Don't forget the suppositories
This is why I never use max effort! I’ll stick with my suspenders, thank you.
That’s the real shape of the problem.
What happens if someone handwrites a Claude output, then someone uses that handwritten text as a reference. Now you've got a watermarked idea which may have no direct linkage to the usage of Claude.
Are you worried about being accused of using LLMs to generate your work? As long as you don't plagiarize you have nothing to worry about.
You can't make a blanket statement like this without knowing how the watermark is implemented.
I'm not too sure about that, people making stuff have already gotten penalized by overzealous AI detectors, most recently Kurtzgesagt.
What if I unknowingly read content written by Claude in various articles and it influences my own writing style?
How is that different from referencing digital text that someone copied and pasted from Claude?
Because there's an expectation of authenticity from the written word. If you've referenced something handwritten, you don't expect it to be the output of an LLM.
Similarly, if you quote someone word-for-word, you wouldn't anticipate their words to be flagged as Claude content, but if someone memorized Claude output word-for-word. That would still be classified as a Claude output.
Going forward you could categorize the influence of Claude on a population based off a percentage match between their spoken words with the LLM prose.
Maybe there is a reason why Opus 5 produces such word salad conversations
It will just thread some load-bearing seams through the paragraphs.
> have some kind of weird pattern baked into their text to act as a watermark.
public abstract class BaseAnimalBeanFactoryGeneratedFromClaudeFactory
Most likely watermark will be proportional to the input/output ratio, i.e. if you input a long document and ask to make edits, it will not attempt to watermark it. On the other hand, if you provide a tweet and ask it to write an article, that will include watermark. Just a guess (and yes, it feels flawed)
Load-bearing==claude
I can tell you how: Claude produces a huge wall of text with jargon ridden bullshit and invented terms no human subject matter expert would seriously use and overuse.
I've heard that this kind of watermarking process works by biassing the statistical sampling towards a partition of the set of possible next tokens (red set and green set), at each position. It might only be a slight nudge each time, but over a sequence of tokens, the likelihood of repeating the bias by chance is increasingly improbable.
The bias is different for each position and follows a defined RNG, seeded somehow predictably.
Can be either an open algorithm, or not. If not open, then an API could be provided to determine if text is watermarked or not.
How it applies to code - maybe it could be a subtle nudge to symbol names, etc, I'm just speculating (I only read about this in passing very recently).
There's a computerphile video (https://www.youtube.com/watch?v=XZJc1p6RE78) with Dr. Mark Pound explaining a paper by John Kirchenbauer, Jonas Geiping et al. (https://arxiv.org/abs/2301.10226) that described a method for watermarking LLM output like this. It's not directly stated anywhere in the Claude support article that this is what they're using, but the properties of the watermark described seem to point to this method.
Based on my understanding, it can only be applied to code in very limited ways: docstrings, variable names, string literals. The code itself can't really have tokens changed to another equally correct token (the foundation of the watermark) because then the code breaks! And the few places that you can do so are likely erased by formatters anyway.
> a defined RNG, seeded somehow predictably
So, an NG?
If it's based on position mod 2, wouldn't inserting or deleting (or splitting/merging) words every now and then trivially defeat it?
If it is based on position mod 2 then wouldn't inserting/deleting (or splitting and merging) words every now and then defeat it?
> "Claude models launched on or after August 2, 2026 support marking at launch."
No Anthropic model has been launched in August.
So my code that Claude makes, which previously was using the best (most probable) tokens for the job, will now be getting worse in random positions, to appease a voluntary EU suggestion. Love that.
They aren't using greedy decoding, there's enough randomness in sampling to swap some with independent signal.
We need to just stop pretending we can reliably tell if plain text is written by an LLM.
It’s just not a reasonable ask.
I have had a hunch for a while now that (in addition to these tools), Anthropic has actually leaned in to Claude's distinctive manner of writing since it makes the text more obviously AI generated and thus less susceptible to misuse.
That's not necessarily the same thing as a markov-style fingerprint but it could be a correlated factor.
I had a similar thought but I assumed they leaned in because it improved performance on coding or something like that
It could also partly be a byproduct of examples of claude writing being in the dataset, which of course anthropic has lots and lots of and they do train on.
no way. there's just no good excuse for why "load-bearing" and "worth flagging" are everywhere now, I've pretty much never seen that in the wild before
I suspect it's because of alignment concerns. The more deeply they can integrate their principles, the harder it'll be to misuse. Or at least that's the idea.
It's pretty trivial to command it to not speak that way. That's one of the first things you should write into the prompt. What style you want it to write in. Make it use a very concise and dry academic style with no overt LLMisms, melodramatic or flowery language, or metacommentary.
Can someone help me understand how exactly this watermarking of text works?
Given that text is, well, text, and not some kind of binary format, I don't see how any watermarking can work unless you insert characters which are invalid under Unicode. I further don't really understand how this won't be perceivable by assistive technology (the "watermark" will just appear as either unreadable characters, or if the watermark is mixed thoroughly enough into the text, it will scramble the text to any speech synthesizer and will make it really really obvious). Thus, I don't see how this wouldn't be insanely trivial to remove. And this is before we get into things being put on the clipboard. Sure, I can press the "Copy" button at the end of each response, but what I can also do is manually select the response and copy it, or only copy partial selections, or any number of other things. How does this "watermark" (or any "watermark" technology) take into account this?
So, really, to summarize this: I see no way of this actually being technologically achievable unless we revise the very core of how computers work and encodings for textual information. So I'm very curious as to how this is actually supposed to work.
Have a look around for token biasing, or green lists. It's based on a nudge to the choice of the next token (which can always be drawn from a set of possibilities which are all probable enough).
At first I thought this approach was just the "LLM flavour" of writing, but it's way more subtle, especially as the bias is applied uniquely for each token position.
Yeah, will do, this sounds interesting since I'm not entirely sure how this would actually be reliable to any degree. Thanks for the help, not sure why I got downvoted since I was genuinely curious.
it's a statistical way. like for example maybe in your above paragraph claude maybe writes "Thus, I don't see how this wouldn't be insanely <easy>(instead of trivial) to remove" and then also says like "And this is before we <analyze> things being put on the clipboard." or maybe the i just says the word "the" in a certain pattern or frequency.
you can then consistently like figure out if it was claude that wrote the sentence. it is easy as you noted if you just get another ai to read it and then rewrite it.
An interesting factor of this is competition.
If Claude was the only model family they could ship a change like this and users who want to cheat (or don't like watermarks for other reasons) would just have to put up with it.
In a world with many different competing models, the risk of losing customers to other providers over this is much more real.
Maybe they've looked at the numbers and the portion of people who clearly use Claude to cheat on examples etc is so tiny that losing them to other providers isn't a problem?
Scott Aaronson spoke about this in a colloquium where he said that this was mooted at OpenAI before the decision was made by Altman to not implement it for the reasons you describe.
https://youtu.be/9udWn1Hlj_s?si=VWOiK5-y4zcyDoHI
OpenAI will soon be adding watermarking to text as well, as it signed the EU Code of Practice on Transparency of AI-Generated Content: https://openai.com/index/advancing-responsible-ai-across-eur...
I’m more worried that this will degrade performance. I want the best results from a model, not the results that fit a constraint that’s not defined by me. Any increased cost or latency is also unacceptable.
There are already small models trained specifically to prevent statistical detection, e.g., https://huggingface.co/kalpeshk2011/dipper-paraphraser-xxl
I guess whoever is the policy maker is assuming that some protection is better than none and that most people will not reach for such tools.
Either that or they want to comply with the EU AI Act when it affects them.
> We’re also working to enable users and other third parties to detect Claude’s embedded watermarks and provenance metadata.
This seems to be similar in execution to Google's SynthID. I hope they release actual code the technically proficient can use, unlike SynthID which can only (afaik) be queried with Gemini's UI.
The moment Google announced SynthID, the first domain I bought was deSynthID.com
Several open-source projects have already proven SynthID to be ineffective.
There are many free lock picking tutorials, but yet locks are still effective.
Seems like an awful idea. I hope that that "watermark" will soon be discovered, reverse-engineered, and that tools to remove it will appear.
I hope all models adopt it.
Thankfully, there are a variety of Chinese models that never will. I think we all know that in a few years, they will also be the only relevant offerings on the market, due to not being bogged down with over-zealous ""safety"" footguns.
Other models will end up diffusing it and making the signal indeterministic and irrelevant.
People with dyslexia and dystrophia, commonly use LLMs to proofread content. Even Anthropic admits this is a limitation.
Yes, I’m audhd and dyslexic.
I am cancelling my Claude max 5x subscription and moving to ChatGPT pro. I have difficulty enough trying to ensure my meaning comes through correctly, along with everything else; to now have to look out for/analyse watermarks too?
I feel shamed enough by society, thanks Anthropic.
Don't you think OpenAI will do this too soon?
How exactly does this impact proofreading? You can manually apply the suggestions (typo here, unnatural sounding sentence there, etc.) the LLM gives you to your own content, and it would stay watermark-free.
Unless with "proofreading" you actually mean having the LLM write your content for you.
People with executive dysfunction too. LLMs bring execution costs down to near zero and are therefore assistive technology.
"This is my emotional support gun. It makes me feel safe despite my CPTSD and is therefore assistive technology."
> Generated text will carry embedded watermarks, and generated files will include digitally signed provenance metadata where supported.
This should make it easier to catch cheaters who use Claude, right? Unless everyone runs their artifacts through some watermark and metadata sanitizer?
As long as they’re in the EU.
From the linked article
> Regions. Marking will apply to output from supported models wherever Claude is offered, worldwide.
> Unless everyone runs their artifacts through some watermark and metadata sanitizer?
It will happen if Claude tampers the text. Guaranteed.
Text is too low bandwidth to classify reliably without lots of false positives. Especially as people start talking like LLMs.
The approach Pangram has taken which works pretty well is to simply lower the recall a lot but ensure the precision is very high. Which means potentially high false negative rate but low false positive rate.
I don’t like the idea of hacking a response to contain a watermark. I also don’t like the idea of false positives detections coming directly from Anthropic. If people read more AI generated content, people will probably start writing more in that style
The amount of times ‘delve’ appeared in general conversation in the last couple of years shows the influence LLMs have on society.
I have no idea why you were down voted for this. Language is alive and people adopt it from sources they hear a lot.
Moreover, what if you quote text that happens to have been generated by Claude, does that bump up the AI-ness score of your source file or document?
The flip side of this is that if AI-generated content becomes reliably identifiable and carries a stigma, then people might deliberately change their styles to be more diverse and human.
One example I've seen are junior employees at my company deliberately adopting a lowercase/less punctuation writing style so as to stand apart from AI.
no thanks.
OpenAI has been watermarking their images with C2PA for some time.
They should just replace the spaces by one of Unicode special space characters.
Can it be circumvented? Of course. Will most people go through the trouble to circumvent it? No.
If it's that simple and obvious, you'll have 10 "Remove Claude Watermark" web-apps by the end of Day 1. Most of them coded by Claude.
Hell, it'll probably happen no matter how sophisticated their watermark is. There's no watermark in text that can't be detected and removed, and no text that can't be converted to generic keyboard ASCII.
You forgot about the cases where (1) people don't care, (2) people want to say "I used an LLM for this". I'm convinced that those cases happen more often than you think. Why not cover them with a simple mechanism? It's also in the interest of AI companies who don't want to train on AI output.
Depends on the pushback in different sets of users. Students for example would clean it up.
Sure, but let's first find out how many % of people are willing to be frank about their AI usage, and/or don't care about it. My guess is it is worthwhile to do this.
But the source codes of those web apps will also be watermarked. /s
Those invisible spaces get wiped by the first sanitizer in any normal ide. Worse it'll instantly break parsing for configs like yaml where spaces are critical for structure
https://unicode-explorer.com/articles/space-characters
U+2800 or U+3164 would be nice.
But as I remove unwanted characters with grep before layout in InDesign, someone will make a skill for removing such space characters.
We already have one, our Claude setup already requires output to be 7-bit ASCII clean and scans it for such.
Wasn't enough to play cat and mouse with ad removal, now we can also do the same with watermarking.
You could flip bits in the font itself, but I'm really wondering how portable this is.
I know you're gonna read this so I'll be blunt. This is bad for your brand.
Reactionary emotional advice does no good, it just makes people want to hold their positions more defensively. If you care enough to say something, care enough to say it with reasons that might shift someone’s perspective.
I expect a “Prettier” for AI generated text in the near future.
This is terrible news given the stigma against AI in general. I really don't want people singling me out for it.
As if "AI generated content" even exists instead of LLMs being a piece of tooling that is directed by a human author.
Does it mean their models will always write slop? Making the writing non-collapsed to specific patterns seems to break any injected/learned fingerprinting.
If the western AI companies are forced to comply with this type of BS, and develop their models to do their job while balancing a book on their head and hopping on one foot, the Chinese models just got a free pass to completely dominate the frontier.
EU regulation does it again!
What's the problem, really? Given the direction the U.S. has been heading in recent years, I wonder what really sets it apart from China. Europe needs to maintain an equal distance from both the U.S. and China.
If the Chinese want to sell to EU customers, they probably have to do the same.
So this won't be happening in the US, but in the EU:
"When a supported Claude model generates text, it weaves an imperceptible watermark directly into the text itself. You won’t see it, and it doesn’t change the meaning, quality, or readability of Claude’s response.
Because the watermark is part of the text, it will travel with the text when it’s copied and pasted elsewhere, and may persist through some editing. Watermarking will be applied at the model level, which means it will be present no matter which Claude product or surface the text comes from."
I would love to see what this looks like in practice. Especially in generated code. I assume this is more than insertion of non visible special unicode whitespace characters, but more in the pattern of the text content itself?
It essentially looks like the difference between two different runs of the model with the same prompt but different seeds. The watermark is essentially a small bias in the model such that when there's multiple different tokens that could conceivably follow the previous token, the model will only pick some subset of them (the subset is derived from a hash of the previous token). This bias can then be checked for statistically (without needing access to the model and without needing the whole prompt), and for longer text where there's enough freedom in word choice you can show that it would be vanishingly improbable to accidentally follow the rules in the watermark.
I'm guessing - probably some textual variation on Benford's law? [1]. Trivial for compute, painful for a human.
- "Ensure distribution of vowels is in >99th percentile of human work"
- "Ensure the distribution of the letter "s" is within 99th percentile of human work"
- "Ensure the distribution of the letter "L" is periodic with periodicity within 5% of 1/N characters.
- "Ensure there is a cross-linguistic 'typo' (colour vs color) at 1/N words, where N: 1000 = Model1, 2000 = Model2, 3000 = Model3.
- "Ensure the distribution of tense error is within 99th percentile of human work"
If more than 3 dimensions have a score >99% percentile of human, let's call it watermarked...
- 1) https://en.wikipedia.org/wiki/Benford%27s_law
Models can't reliably follow instructions involving their own logprobs unless they can take agentic control and use quite sophisticated dynamic grammars/structures/constraints to force this behavior in one shot (which can be slow and the dynamic grammar modification feature isn't supported in closed model APIs for safety reasons) or repeated attempts at rewriting which is expensive/slow.
Yes they can do this, but it's more likely closer to the original "red token, green token" paper: https://arxiv.org/abs/2301.10226
i.e. take half of your LLMs vocabulary, and upweight its probabilities by ~55% to the other half's ~45%, and scan for overuse of this half of all tokens. You can even choose a different half/slice for every individual user, for every individual action. You can implement this under the hood cheaply with logit-biasing.
I would hate to have any of these rules effecting my output
IIRC, watermarking text could be as simple as training the model to use specific words/phrases more frequently than what you would expect to find in human-written text, to the point where it's highly statistically improbable that it wasn't AI generated. I assume similar logic could apply to code in the form of functions/code styling.
That's probably an over simplification. Also a solid defence that can be used against complaints about the way AI writes text.
non visible text is extremely easy to filter with a git hook, a post tool call hook, or just a script. I doubt they are doing that
Or grep, in a skill. /clean-cc-watermark just entered the chat…
Article specifically says "worldwide"
I agree with your reading, I initially misread it.
Good.
They should make it easier, to detect slop so we can ignore it quickly.
I hope Pangram makes an API or an extension to analyze a page to detect slop on a page and then closes the tab immediately.
Nobody should be wasting time on garbage LLM output in code, text, image and videos.
Panagram is a scam.
(This is the part where you provide extensive extraordinary evidence to your claim)
Scam might be too strong a word but it certainly has far higher false positive rates than they are claiming, and their output is at best misleadingly presented: https://freddiedeboer.substack.com/p/i-wouldnt-say-pangram-i...
No, they are the ones making claims, especially their CEO saying things like a 1/10000 false positive rate. Their own testing showed a 2% rate, which is insanely high when you talk about the number of papers students turn in. Worse their testing methodology compared it with pre-llm documents and not post llm documents that were human written (much harder and more expensive to verify), by treating language as static.
Pangram is subjectively very useful and I personally subscribe, but the burden of proof is on them. The product is very much "trust me bro" and I fear that if they ever try to improve recall both their precision and reputation will tank.
Yet another reason to support open-weight alternatives, I guess.