> My initial speculation was that maybe they’d hide invisible non-printing Unicode characters in the text.
Or just em dashes? /s
Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
241–250 of 776 posts
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#242Heck. The entire process itself of LLM text generation is a perversion of writing.-
Further heck: It can be said it ain't even writing.-
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#243Earlier quoted context omitted.
It seems to me like he started out mad and looked to justify it. I'm skeptical that anybody generating LLM text is really all that concerned about optimal word choice. Or even particularly good prose. But let's pretend that person exists. If that person tried, say, an open model and that same model with watermarking applied, I'd be eager to hear their thoughts on the prose quality. Especially if they built an experim…
Google has A/B tested watermarking on millions of responses. They say they observed no difference in user behavior.
It says they observed no difference in people clicking thumbs up or down. There are loads of other behavior that they didn't observe; like, say, switching to a different LLM.
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#244I recently found there was no website covering a historical subject, with sites begging to be visited. The information was scattered and not presented in the form people would find most useful (a map).
A couple of days later it was up with the most comprehensive review of the available info summarised and referenced ready for a human researcher to explore.
This was only possible due to AI. It would have taken me weeks to chase it down and summarise it, so it would never have happened. Let's get off our high horse about AI writing.
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#245> I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo ran…
The quality of the LLM just _is_ the quality of the token probabilities it generates. Better quality token probabilities, better quality output. Worse quality token probabilities, worse quality output. Watermarking changes the probability calculations for reasons other than quality. It can't not compromise quality. It literally leads the LLM to occasionally chose different tokens just for watermarking purposes.
No one uses a pure random function over the whole probability distribution described by the LLM's output. For example, there is exactly 0 probability that the chosen next token by any common API or even local LLM runner would be a token whose final value is "0.0001" if there exist at least K tokens whose value exceeds "0.7".
Also, as long as the same sampling strategy is used during training as the one used during inference, then the LLM will actually do much better with the biased sampling strategy than it would with a fair one - because that is what it was trained to optimize.
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#246My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ...
So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illicit means.
Any university using AI detection in their submission pipeline, or lawyers, editorialists, proofreaders that check for AI marks will be sending significant amounts of text like unpublished research, books, potentially internal documents and more, most of which is high quality human written, to dozens of AI companies, blindly trusting they won't train on any of that.
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#247I think we will see a hidden motivation behind this as ultimately so the output can have an author, the author can be attributed and finally the output will be copyrighted and so the LLM has more value.
Everyone who invests in AI companies wants to see the value of their investment increase.
I'd give it about 3-5 years until an AI company claims copyright over code their LLM produces. This is a crucial step in that path.
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#248> I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo ran…
This is true and the author seems to not understand the problems with greedy (top 1) decoding or the fact that watermarking affects only high entropy tokens.
But the published watermarking methods still have a slight negative effect on perplexity, so there is something more to it.
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#249> The provider must mandate in their terms-of-service that users not remove the watermarking. So, you don't own the generated text, and can't use it freely then. What if I copy paste a section, or rewrite a section of text to my liking? What if I rewrite some lines of code that contains the mark? Security theater, and vague enough to be used as a weapon against who the government wishes. I hope it's left off for non-…
You know you can just ignore EU laws outside of the EU
They are literally asking for Anthropic to do that rather than roll out 1 system for everyone.
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#250> Someone with the secret key can determine which list a word will be on at each token generation point (which is how the watermarking is detected); those without the secret key cannot. How is this supposed to work in an actual lawsuit? Will Anthropic offer some sort of tool / (paid?) webservice to check for watermarks using that "secret key", and a judge is supposed to just believe whatever that tool's verdict is? A…
Yes, exactly, that is my understanding as well. Since the watermarking is based on a symmetric key by design (as to not be easily able to test and remove the watermarks), the providers will need to offer a watermark detection API.
I see even more problems with this. To check any text for those watermarks, it needs to be sent to dozens of AI companies to check, potentially paying them all for just determining whether it matches their watermark, and more concerningly sending all that mostly human-written, often high quality text like unpublished research or books, to AI companies that almost all proved to obtain training data through all kinds of dubious ways.