I wonder if this will become a new revenue stream for providers. Want to know if Claude generated some text? There's a free web form you can paste into.
Of course, you can also run your own check service if you pay some API fees, and those checks can be a lot more convenient for users since the services can check multiple sources, to whom they are paying for the privilege.
Then someone washes the text through a local model that rewords it, the markers are lost, amd they're clear again.
I can see this being real useful when they scrape the internet for content. And they don't want their own crap. Or just to see how prevalent their stuff is.
I wonder if this is for use in future legal battles over ownership/creation/invention of new software and/or concepts.
For example, someone inventing a new technology might use AI to assist with code prototypes/doc prep etc. Where would that leave the rights of the owner/creator/inventor. These text watermarks provide fuel for legal battles.
I think the goal is not to validate if you used Claude, the goal is to insert enough information in your code to identify who you are over at Anthropic, like a covert unique hash to identify you, hidden in plain sight, but that's my own speculation and not based on anything.
I'm having a hard time thinking of any other use case... What other reason is there to fingerprint your code?
I assume if they arrest you for vibe coding something that violated computing laws (hacking) they can then scan your code, send it to anthropic, anthropic confirms that Claude was used on your account, on x device to build malware.
Remember they busted a hacker because of his Windows unique install ID.
The purpose be to claim in court, ownership and/or rights over said works of an individual who has used AI to assist in the production of the output. Even if the actual idea or concept comes from a person, if the AI inserts watermarking into text, the legal fuel is the watermarking of text, which can then be used in court to fight for rights, where otherwise there would not even be any debate.
One aspect that people seemingly aren't talking about is the impact this has in the model's creativity. Because the model will nudge each word towards group A vs group B, you're losing on creativity, especially more so if the nudge isn't a gentle 55% but something like 70% or 80%. So essentially they're forcing the model to be less creative for the upside that the longer the text the easier it is to detect the watermark.
This is even worse in such forms of writing like coding, where there's even less choices the model can make on what the next token should be. Plain text in code will obviously be watermarked, that includes comments. But the code itself might get watermarked by choosing certain code over others more often.
I'm inclined to believe the models will be instructed to not watermark code, especially since it's harder to detect reliably because the shorter the body of text the harder it is to detect, but who knows what Anthropic and all the other AI labs will decide to do in the future.
EDIT: Also for those in the comments who are naive enough to think Anthropic is doing this just because the EU said so and not because it's beneficial to them (and all other AI labs), well, you are indeed naive. Identifying code will be paramount in training future models because the more synthetic data you feed it, the more cannibalization happens, the worse the models will perform over time due to lack of good data, among other such reasons as selling AI detection services to colleges, and a plethora of other reasons.
As per the link, the words in the green and red groups are calculated dynamically, so it's not like the model is going to be told "use 'unique' over 'unusual'" and suddenly writing from the model will contain the word 'unique' far more often than 'unusual'. So I'm not sure it's clear that this has an impact on creativity as such?
That being said, I do question how this will apply to code as opposed to prose. Even data dense text (ie, if you ask Claude to evaluate what running shoe to buy, and it spits back a list of options with reviews and prices) may struggle.
What it probably will work well at it flagging the current tsunami of entirely AI generated novels on Amazon/Kindle, which is...honestly not without value.
> Identifying code will be paramount in training future models
True, but note that this strictly allows providers to identify text generated by their own models. If Anthropic wants to filter out GPT generated text in their training data, they'll need to feed it through an OpenAI API, which is implausible. So it might help on the margins, but I don't think it solves the problem of model collapse.
It is worse than that, it can make text unintelligible. I have Claude explain what is happening in a PR, and the technobabble and use of rare words have to look up in a dictionary make it difficult to understand. I need another LLM to translate what the LLM is saying.
And I pray it is because of the EU AI Act, because it is worth giving them my personal information to prove I am not in the EU so they can turn this off. Heck, I will pay more to avoid this crap.
If it were not so noticable, I would shrug it off. But it has made things clearly worse this year.
I can't believe I got this in one. I guessed a math PhD interested in steganography could come up with a good method in a week, and an idiot like me figured it out in 5 minutes.
This is a fun challenge, I think: make the AI generate text with no obvious AI shibboleths. I haven't found a good way to do it at all yet, the more you ask it not to, the more you lower the output quality, which paradoxically makes it more sloppy, not less. Funnily enough I think "lower-power" LLMs are much better at this, like Haiku is much better at keeping voice on track with those kinds of instructions.
I have done it for a long time now. It's actually pretty straight forward to do it, but you have to run it in a pipeline, and you have to add deterministic checks as well.
There was an odd key to unlocking it sounding like a human. I spent like a month on this topic last year, and I randomly stumbled upon something that ended up massively improving generated tests.
It was a shot in the dark after failing so often. And then it suddenly sounded good.
okay so text generated in America by and American AI will be watermaked to make Brussels happy? and then (only) elite companies will have access to some portal to they can label text as AI generated?
New job idea: have a human reword/summarize and manually input AI output to remove the watermarking.
I wonder if this will become a new revenue stream for providers. Want to know if Claude generated some text? There's a free web form you can paste into.
Of course, you can also run your own check service if you pay some API fees, and those checks can be a lot more convenient for users since the services can check multiple sources, to whom they are paying for the privilege.
Then someone washes the text through a local model that rewords it, the markers are lost, amd they're clear again.
I can see this being real useful when they scrape the internet for content. And they don't want their own crap. Or just to see how prevalent their stuff is.
I wonder if this is for use in future legal battles over ownership/creation/invention of new software and/or concepts.
For example, someone inventing a new technology might use AI to assist with code prototypes/doc prep etc. Where would that leave the rights of the owner/creator/inventor. These text watermarks provide fuel for legal battles.
Am I wrong?
I think the goal is not to validate if you used Claude, the goal is to insert enough information in your code to identify who you are over at Anthropic, like a covert unique hash to identify you, hidden in plain sight, but that's my own speculation and not based on anything.
I'm having a hard time thinking of any other use case... What other reason is there to fingerprint your code?
I assume if they arrest you for vibe coding something that violated computing laws (hacking) they can then scan your code, send it to anthropic, anthropic confirms that Claude was used on your account, on x device to build malware.
Remember they busted a hacker because of his Windows unique install ID.
The purpose be to claim in court, ownership and/or rights over said works of an individual who has used AI to assist in the production of the output. Even if the actual idea or concept comes from a person, if the AI inserts watermarking into text, the legal fuel is the watermarking of text, which can then be used in court to fight for rights, where otherwise there would not even be any debate.
I dunno, maybe they want to be able to identify slop and exclude it from training sets?
One aspect that people seemingly aren't talking about is the impact this has in the model's creativity. Because the model will nudge each word towards group A vs group B, you're losing on creativity, especially more so if the nudge isn't a gentle 55% but something like 70% or 80%. So essentially they're forcing the model to be less creative for the upside that the longer the text the easier it is to detect the watermark.
Here's a visual representation of the watermark: https://i.imgur.com/JNUIykX.png
This is even worse in such forms of writing like coding, where there's even less choices the model can make on what the next token should be. Plain text in code will obviously be watermarked, that includes comments. But the code itself might get watermarked by choosing certain code over others more often.
I'm inclined to believe the models will be instructed to not watermark code, especially since it's harder to detect reliably because the shorter the body of text the harder it is to detect, but who knows what Anthropic and all the other AI labs will decide to do in the future.
EDIT: Also for those in the comments who are naive enough to think Anthropic is doing this just because the EU said so and not because it's beneficial to them (and all other AI labs), well, you are indeed naive. Identifying code will be paramount in training future models because the more synthetic data you feed it, the more cannibalization happens, the worse the models will perform over time due to lack of good data, among other such reasons as selling AI detection services to colleges, and a plethora of other reasons.
As per the link, the words in the green and red groups are calculated dynamically, so it's not like the model is going to be told "use 'unique' over 'unusual'" and suddenly writing from the model will contain the word 'unique' far more often than 'unusual'. So I'm not sure it's clear that this has an impact on creativity as such?
That being said, I do question how this will apply to code as opposed to prose. Even data dense text (ie, if you ask Claude to evaluate what running shoe to buy, and it spits back a list of options with reviews and prices) may struggle.
What it probably will work well at it flagging the current tsunami of entirely AI generated novels on Amazon/Kindle, which is...honestly not without value.
> Identifying code will be paramount in training future models
True, but note that this strictly allows providers to identify text generated by their own models. If Anthropic wants to filter out GPT generated text in their training data, they'll need to feed it through an OpenAI API, which is implausible. So it might help on the margins, but I don't think it solves the problem of model collapse.
It is worse than that, it can make text unintelligible. I have Claude explain what is happening in a PR, and the technobabble and use of rare words have to look up in a dictionary make it difficult to understand. I need another LLM to translate what the LLM is saying.
And I pray it is because of the EU AI Act, because it is worth giving them my personal information to prove I am not in the EU so they can turn this off. Heck, I will pay more to avoid this crap.
If it were not so noticable, I would shrug it off. But it has made things clearly worse this year.
Unfortunately, that's unrelated; only models released after August 2nd support it so far, according to Anthropic.
I wonder how much output text it would take to work out the key that the watermarker is using?
"Whoever gets to lean on how the dice land"
I'm a native English speaker and I have no idea what this means.
I can't believe I got this in one. I guessed a math PhD interested in steganography could come up with a good method in a week, and an idiot like me figured it out in 5 minutes.
From now on, a programmer can no longer claim that they wrote the code themselves when it was written by AI, in a way that could bypass this issue.
AI writing at its finest
They/the LLM made sure to replace the em dashes with semicolons
This is a fun challenge, I think: make the AI generate text with no obvious AI shibboleths. I haven't found a good way to do it at all yet, the more you ask it not to, the more you lower the output quality, which paradoxically makes it more sloppy, not less. Funnily enough I think "lower-power" LLMs are much better at this, like Haiku is much better at keeping voice on track with those kinds of instructions.
I have done it for a long time now. It's actually pretty straight forward to do it, but you have to run it in a pipeline, and you have to add deterministic checks as well.
There was an odd key to unlocking it sounding like a human. I spent like a month on this topic last year, and I randomly stumbled upon something that ended up massively improving generated tests.
It was a shot in the dark after failing so often. And then it suddenly sounded good.
Ex:
> The one idea in this step: a model writes by rolling weighted dice between several words that would each be fine.
Telling me there's just one idea is taking up even more space than just saying it.
okay so text generated in America by and American AI will be watermaked to make Brussels happy? and then (only) elite companies will have access to some portal to they can label text as AI generated?