Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

421–430 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#421

Post by eloisant »

mangoman wrote: post #414
voidUpdate wrote: post #199

> "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much

What are you talking about? I understand OP has the perspective of writers, but let’s say you’re asking an LLM to explain a concept and it uses green words that are actually more difficult to understand. Or you ask for an analogy to explain a topic and the analogy doesn’t quite land because the description used “gray” instead of “overcast”.

Watermarking or not, LLM are already using RNG to pick a variant between different words/expressions.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#422

Post by Majromax »

vintermann wrote: post #379
unbelievr wrote: post #361

Earlier quoted context omitted.

The paper for it is open. The technique isn't really hiding information in the text itself, but by forcing some of the rolls to follow a specific pattern. LLMs work by estimating the most likely next token, so there's sometimes a list of possible candidates that would all work in the text (e.g. synonyms). At low "temperature", the output is a bit more deterministic and otherwise it's a weighted dice roll of which tok…

A way to think of it is that any time the model faces a choice, it leaks some fractional bits of information. Instead of making the choice randomly, you can put information in those bits. But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty. Exotic prompts ("Use no more than five E's in four consecutive words anywhe…

> But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty.

When you only need to encode one bit, the signal to noise ratio can be very low. If I try to write my own human words under the policy of "try a little bit to avoid the letter 'e' in every fifth word," then a sufficiently long text would still be 'watermarked' even if I only succeed in this dictum (e.g.) 10% more often than the baseline.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#423

Post by unclebucknasty »

ThePhysicist wrote: post #338
ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

Would really like to know how their watermarking technique works, and if they can use it to store arbitrary information in the text (I assume they can if only in a limited way). I assume if the text is long enough they could add all kinds of metadata that would then be undetectable as the model that generated the text is the "cryptographic" key that encodes the data. I wonder if you can have another model run over th…

Simple version: In instances wherein the otherwise statistically chosen next word is a "toss-up", watermarking removes the randomness by imposing specific choices, determined by a key. This then becomes a detectable pattern when scanned with the key (stastically—detection itself is probabilistic).

>use it to store arbitrary information

No additional data is embedded. The range of available data is constrained by the text being generated (i.e. the sets of "next words" per text).

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#424

Post by doginasuit »

voidUpdate wrote: post #199

> "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much

Do people not realize this will apply to ALL Claude output, not just writing you ask it to produce?

> Marks will apply to output from supported Claude models across Claude Platform (API), Claude, Claude Code, Claude Cowork, and Claude Tag, and wherever Claude is offered.

I've never asked an LLM to generate writing I wish to post as my own. I don't understand why we think it is a good idea to fudge all output just so people can't cheat on their homework or generate slop. It won't have any impact on those things because there will always be models that don't do this. All it will do is increase the rate of false negatives.

It is deeply misguided regulation and Anthropic should have just said no on grounds of common sense.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#426

Post by Majromax »

vintermann wrote: post #386
ddalex wrote: post #356

Earlier quoted context omitted.

Probably they bias the RNG for selecting the next token. This can be done practically in a lot of ways, including during training. I suspect the signal will be significantly under the noise floor, so it's not detectable if you don't know exactly what to look for, but certainly you can submit more information then the textual contents.

But if the user's prompt is in the context, you don't know exactly what the RNG chooses between. I don't know what trick they use to get past that, but it seems impossible to get by it in the general case (i.e. if the prompt can be anything) and you'll probably quickly compromise quality if you try.

A reasonable guess about the algorithm is 'A Watermark for Large Language Models' (https://arxiv.org/abs/2301.10226). The idea is that each generated token (or bigram) seeds a strong PRNG that splits the vocabulary into a 'green' and 'red' set. The sampler then tries to select a 'green' next-token for generation.

After-the-fact checking only needs the vocabulary splitter, which is independent of the LLM. Over a sufficiently large text non-watermarked text would expect to use green and red tokens with the baseline probability, and that difference can easily become statistically significant over sufficiently long texts.

The basic algorithm has obvious knobs to tune, among them the initial ratio of red to green tokens and how hard the sampler tries to pick a green token. These would balance fidelity to the original distribution against watermark detectability (minimum required content length for statistical power).

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#427

Post by ttoinou »

Majromax wrote: post #422
vintermann wrote: post #379

Earlier quoted context omitted.

A way to think of it is that any time the model faces a choice, it leaks some fractional bits of information. Instead of making the choice randomly, you can put information in those bits. But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty. Exotic prompts ("Use no more than five E's in four consecutive words anywhe…

> But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty. When you only need to encode one bit, the signal to noise ratio can be very low. If I try to write my own human words under the policy of "try a little bit to avoid the letter 'e' in every fifth word," then a sufficiently long text would still be 'watermarked'…

Whereas if you fully avoid the letter e, everyone will know you are George Perec

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#428

Post by mangoman »

I’m surprised by the comments here being so favorable to anthropic. The comments are right about there being no “best” token, and yeah Gruber may have an agenda here. But I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. Take the “gray” and “overcast” choices. And lets say before applying synthid the percentages were 48% and 52%. Those percentages were learned from the training data and RL. To change those percentages to 45% and 55% in a non learned way makes it seem like training wasn’t important? Or more likely they dont have the data that shows the failure modes? also, don’t these choices compound the changes to the distribution in later sampling choices?

It is a bit of a mystery to say that “its okay to choose different tokens that we would have for watermarking bc people don’t notice” as though word choice doesn’t matter. If it doesn’t matter, doesn’t that mean that intelligence is more of a commodity than they would want it to be?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#430

Post by NitpickLawyer »

dofm wrote: post #336
NitpickLawyer wrote: post #283

Earlier quoted context omitted.

> He is not using AI That's ... even worse? So we're all here in the comments trying to figure out what the author means, and what their overall point is, while clearly they don't even use the damn thing? Oof... What a waste of time for everyone involved.

Why? I really don't understand this. Why can't a tech writer take a deep but neutral intellectual interest in something? Isn't it important that some do? Do people have to be stakeholders or clearly on one given team, pro- or anti-, for their opinion to matter? Is it that tribal? It seems fully logical to me that someone who writes for a living (who, as it happens, developed the very markup language LLMs use for ever…

> Why? I really don't understand this. Why can't a tech writer take a deep but neutral intellectual interest in something? Isn't it important that some do? Do people have to be stakeholders or clearly on one given team, pro- or anti-, for their opinion to matter? Is it that tribal?

I don't think it's weird to ask that people commenting on x doing y with tech z at least try tech z, no?

Imagine this pamphlet: Basting a steak in a cast iron skillet with butter and herbs is a perversion of grilling a steak on charcoal. Signed, a life-time vegan who hasn't cooked a steak in their life.

Then imagine people jumping in the comments to discuss. Isn't it a waste of time?

Post reply on HN