Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

611–620 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#611

Post by alasdair_ »

bsenftner wrote: post #374
unbelievr wrote: post #361

Earlier quoted context omitted.

The paper for it is open. The technique isn't really hiding information in the text itself, but by forcing some of the rolls to follow a specific pattern. LLMs work by estimating the most likely next token, so there's sometimes a list of possible candidates that would all work in the text (e.g. synonyms). At low "temperature", the output is a bit more deterministic and otherwise it's a weighted dice roll of which tok…

In practice, it is theater. Are they going to do this with the code output too? This is nonsense security theater for the low thinkers to have a sense that someone is in charge. When we all know nobody is in charge, anywhere.

They say in the paper that they will not do this for code but it may happen for comments.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#612

Post by recursive »

Majromax wrote: post #422
vintermann wrote: post #379

Earlier quoted context omitted.

A way to think of it is that any time the model faces a choice, it leaks some fractional bits of information. Instead of making the choice randomly, you can put information in those bits. But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty. Exotic prompts ("Use no more than five E's in four consecutive words anywhe…

> But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty. When you only need to encode one bit, the signal to noise ratio can be very low. If I try to write my own human words under the policy of "try a little bit to avoid the letter 'e' in every fifth word," then a sufficiently long text would still be 'watermarked'…

There's more to it than encoding one bit. You also want to avoid false positives. You can encode a single bit by XORing all the bits in the UTF-8 encoding. But then you get a lot of incorrect hits. The lower your tolerance for false positives, the more it acts like you're actually requiring more bits in your payload.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#613

Post by alasdair_ »

kfse wrote: post #488
pavlov wrote: post #388

Earlier quoted context omitted.

Their AI model tends to write a lot of lengthy comment blocks. That's a fine place to put the watermark to track those users who accept the code blindly and don't delete/edit the comments.

The model doesn't particularly "know" which tokens within its output are "comments", it's not so easy

It knows when several tokens are about equal vs. times where one token is vastly preferred. In the latter case, that’s usually code or math or something similar and so it won’t alter those tokens.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#615

Post by NewsaHackO »

inigyou wrote: post #125
demibabs wrote: post #107

Earlier quoted context omitted.

Part of the legislation requires them to make a public AI text detector (ala GPTZero I assume). Wouldn’t having that be enough to eventually reverse engineer the key?

Probably not to get the key, but you could certainly use it adversarially to remove the watermark. Removal may come down to changing every third token to a different one.

Yea, they are going to have to monitor how often similar writing pieces are being submitted, or limit the access to the checker somehow.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#616

Post by [deleted] »

ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

[deleted]

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#617

Post by Akranazon »

> I want any LLM I use to choose the very best, most precise words at every single decision point.

No, you don't. If you wanted that, you would set the temperature parameter to 0. But that would lead to less desirable results, not better. LLMs do not set the temperature to 0; they typically set it 0.4-0.7.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#618

Post by Spooky23 »

sfink wrote: post #609

I agree with the substance of this article, and disagree with the author's reaction to it. The part I agree with: It is true that watermarking can be done by "just" swapping one PRNG for another, and it is even true that with today's LLMs, it is possible that this will not degrade the output. But it has a cost, and as things improve, that cost will matter. You are intentionally reducing the degrees of freedom in the…

> So I agree that the watermarking has a cost. But you can't leave out that it is an attempt to reduce negative externalities of AI. Whether it's a realistic or worthwhile attempt is a whole other debate (and Gruber does a good job of debating just that in the latter part of the essay), but saying that the user's needs are the only thing that should ever be considered is reprehensible.

Computers are tools that exist to serve. Creating some bizarro process where we are compromising the technology in service of it's owner to achieve some nebulous goal is gross.

Anthropic is crowing about this achievement because they are afraid of the dirt cheap AI models coming out of China and eventually other places impacting their valuation. Full stop. There's some vague notion of preventing harm without any backing, but a very real cost for startups to develop a compliant watermarked AI model.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#619

Post by sfink »

josephg wrote: post #440
mangoman wrote: post #428

I’m surprised by the comments here being so favorable to anthropic. The comments are right about there being no “best” token, and yeah Gruber may have an agenda here. But I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. Take the “gray” and “overcast” choices. And lets say before applying synthid the percentages were 48% and 52%. Those pe…

I’m sure it matters. But how much does it matter? How much (perceived) intelligence would you be willing to sacrifice for an accurate AI predictor? I’d sacrifice a few %, easily. Maybe 10%. The models are getting smarter at such a fast rate that I’d be willing to lose a month or two of progress to help slow down the AI cheating epidemic. It sounds like you expect this fingerprinting approach would dramatically reduce…

In order to do anything about the cheating epidemic, this has to be robust to adversarial attacks. Color me very skeptical on that one. It seems like reprocessing a generated text and adjusting the distribution is a way easier problem than generating the text in the first place.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#620

Post by vintagedave »

sfink wrote: post #609

I agree with the substance of this article, and disagree with the author's reaction to it. The part I agree with: It is true that watermarking can be done by "just" swapping one PRNG for another, and it is even true that with today's LLMs, it is possible that this will not degrade the output. But it has a cost, and as things improve, that cost will matter. You are intentionally reducing the degrees of freedom in the…

> This is the argument that the world is beholden to my interests. It says that worrying about negative externalities is immoral.

I agree with you, in that the benefit of the commons (preventing damage AI is doing to the world etc) is better than one individual's 'right' to a perfect product.

But I don't think Gruber is saying quite this. He's worried about genuine semantic and intelligence loss, using the example of swapping two words:

> The semantic difference between banana and pineapple is just as noticeable to the human eye as the taste of the two are to the human tongue

Further, is this really a useful way to reduce the damage of AI? It only appears to work if someone already believes and then proactively checks if text is produced by AI. It doesn't really address the core issues.

Post reply on HN