Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

521–530 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#521

Post by talljeff68 »

tosh wrote: post #349

any watermarking ai researchers who can explain this? what if the llm should - repeat something verbatim (important in a compaction prompt) - there is just one correct order of tokens for a somewhat long chain (a certain sequence of control signals) - provide a diff of 2 inputs without punctuation or whitespace wiggle room? how does the drifting work? does it postpone the drifting and drift stronger later? what if ma…

The Anthropic annoucement addresses lots of this. They give an example of generating code where there is clearly a requirement to follow a specific syntax and therefore the watermarking will be much less effective and likely require much larger sample of generated content to build statistical confidence in the validity/existance of a watermark. For code, it is the generated comments that will be more likely/able to contribute to the watermark confidence.

The key to understanding the watermark technology is to realize that the model is/was already using randomness to select among the top most probable tokens, often randomly picking between choices of [nearly] equal weight. The watermarking does NOT change the distribution of the random number generation nor does it affect the range of probabilities for which tokens are being considered. Instead, it only drives the sequence of the random numbers such that they form a cryptographically generated known ordering pattern that is determined by the secret key generating the pseudo-random sequence.

As a result of the approach, assuming inference is being done with all other parameterization of token selection being the same, there should be NO impact on the output quality....the amount of variation of output is within sample of the variation in output that already exists run-to-run of the same prompt. FWIW too: Google has confirmed this experimentally as well through full scale tests and evaluation of online Gemini output in search result pages.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#522

Post by kergonath »

summarybot wrote: post #505
nemo wrote: post #376

Earlier quoted context omitted.

>A lot of people use Claude as a friend/therapist/romantic partner People developing a para-social (pseudo-social?) relationship with a corporate robot have far bigger problems than the word-chooser in their robot "friend".

Very intelligent people need intelligent-others to bounce ideas off of, and the LLM can be that.

Very intelligent people should be able to grasp that a LLM is not intelligent.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#523

Post by ragequittah »

Timshel wrote: post #352
UqWBcuFx6NV4r wrote: post #339

Earlier quoted context omitted.

> Humans reading texts historically come with little negative consequences, but machines reading and processing texts en masse is more dangerous and should be regulated. Citation needed. This is sounding tautological.

Not really hard to find example: License plate ALPR like Flock ?

And not really hard to find examples of humans reading text being way more consequential than that. See: the Bible, nazi propaganda, etc.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#524

Post by kergonath »

deng wrote: post #454
voidUpdate wrote: post #199

> "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much

I'm sure he honestly appreciates the pushback.

I am sure he is quite happy to have people disagree with him on HN. He usually wears it as a badge of pride. We might even have a follow-up in a couple of days about how these techy weirdos lost the plot.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#525

Post by demibabs »

unbelievr wrote: post #361
ThePhysicist wrote: post #338

Earlier quoted context omitted.

Would really like to know how their watermarking technique works, and if they can use it to store arbitrary information in the text (I assume they can if only in a limited way). I assume if the text is long enough they could add all kinds of metadata that would then be undetectable as the model that generated the text is the "cryptographic" key that encodes the data. I wonder if you can have another model run over th…

The paper for it is open. The technique isn't really hiding information in the text itself, but by forcing some of the rolls to follow a specific pattern. LLMs work by estimating the most likely next token, so there's sometimes a list of possible candidates that would all work in the text (e.g. synonyms). At low "temperature", the output is a bit more deterministic and otherwise it's a weighted dice roll of which tok…

> The system is fragile, and small edits (or rewrites by a local model) can fool the detection.

Well no, small edits wouldn’t fool the detection as long as the seeding only uses a small run of previous tokens.

And yeah full rewrites breaking it is by design. The watermark is just meant to tell you whether the text was generated by a watermarked model, not whether the ideas came from AI or something like that.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#526

Post by Alex3917 »

ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

> checking any text for watermarks requires sending the entire text to Anthropic

Couldn’t it be checked in the TEE using confidential computing to keep Anthropic’s algorithm secret?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#527

Post by kergonath »

kgarten wrote: post #384
voidUpdate wrote: post #216

Earlier quoted context omitted.

I'll be interested to see if people can actually pick out which text is watermarked and which isn't once they introduce it. It won't switch "bananas" to "airplanes". It'll switch "I really enjoy eating bananas" to "I love eating bananas" or similar

That’s my point of the post … I had the feeling that the English text editing skills of Claude went significantly down in August. I was frustrated at first not knowing what they are doing. After I read this post, (seeing they introduced it in August) I think it has to do with the watermarking. Try it on a paragraph … the connections between sentences feel clunky now. I will play with it more and see if that’s really…

> After I read this post, (seeing they introduced it in August) I think it has to do with the watermarking.

The power of confirmation bias…

> Try it on a paragraph … the connections between sentences feel clunky now.

We might have a definitive explanation at some point, but there are about a dozen possible reasons for something like this. For starters, is this something really significant and not something you notice because you are looking for it (again, confirmation bias)? A bit like some people still lose their minds when they see a dash, even though statistical analysis showed that they are not a significant marker of AI-generated text?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#528

Post by nathan_compton »

syrrim wrote: post #9

> I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo ran…

I cannot believe the markdown guy is this deranged.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#529

Post by iainmerrick »

summarybot wrote: post #507
iainmerrick wrote: post #484

Earlier quoted context omitted.

They do exactly that "which is the better answer?" test -- I've seen it pop up a few times.

Isn't "which one is watermarked?" a different question than "which one is better?" "Which diamonds are shinier, the blood diamond sourced ones or the ethically sourced ones?" ... that's not the same question as "which diamonds are blood diamonds" (to employ an extreme analogy) Concluding that no one could detect which ones were blood diamonds because they were "equally shiny" is not really correct now, is it?

That's true, but you don't typically explain what you're testing in this sort of (presumably) randomised trial.

And the Daring Fireball article does complain that watermarking will reduce quality. If that's what you're trying to check, "which is better?" is the right question.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#530

Post by kergonath »

jorisw wrote: post #509
nicbou wrote: post #322

Earlier quoted context omitted.

Yep, I literally ask it to say it in 20 different ways. Sometimes a sentence structure or a combination of words will just work better. The goal is usually to simplify a sentence without losing meaning. I don't expect the LLM to read my mind. The unit of work is too small for intent to matter, and I'll just steer the next recommendations in a direction as needed. Most of the suggestions are crap, but they can contain…

And this beats just writing the piece yourself?

I can’t speak for them, but I do this occasionally. Every now and then among the propositions there are one or two that did not come to my mind and that are actually quite good. I still write the whole thing myself, I just ask for advice occasionally.
Post reply on HN