Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

411–420 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#411

Post by daishi55 »

thatjoeoverthr wrote: post #212
dragonwriter wrote: post #15

Earlier quoted context omitted.

That's inaccurate in two ways: (1) The behavior that is approximately what you describe is not "fundamental" (though it may not be something you can disable on some hosted providers), it is an option that is not fundamental (and with runtimes where you have full control can be either disabled or tuned in a large number of manners), and (2) The actual behavior that is approximately what you describe already usually in…

1) we’re not discussing those systems. We’re discussing a chat AI product called Claude, which does not offer those knobs. 2) Claude’s PRNG having a P is immaterial

Claude has those knobs, they are just not exposed to the user. They could make Claude nearly completely deterministic if they wanted to (of course it would be a far inferior product then. But they could).

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#412

Post by LogicFailsMe »

I have no doubt they can benchmax their way into believing the differences caused by watermarking are imperceptible. But I'm skeptical there won't be a drop in quality that puts them at a disadvantage relative to competitors that don't do this. This adds a new constraint changing predicting the next token to predicting the next token that carries the watermark. It's hard to see how this can avoid making the watermarked output <= the desired output in quality.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#413

Post by masswerk »

Mind that even in their first example, "The results of the study were quite (important | significant | substantial | notable)", the meaning is by no means interchangeable.

"Important" refers to impact, "significant" to the statistical qualities of the underlying hypothesis, "substantial" to the work involved, and "notable" is a referential judgement by the speaker. The implied normalization of words and their respective meaning also marks one of the mechanisms how "slop" is typically creeping into the productions of "broad verbose interchange replicas" (it's all interchangeable, and a choice isn't really that, a choice, isn't it?).

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#414

Post by mangoman »

voidUpdate wrote: post #199

> "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much

What are you talking about? I understand OP has the perspective of writers, but let’s say you’re asking an LLM to explain a concept and it uses green words that are actually more difficult to understand. Or you ask for an analogy to explain a topic and the analogy doesn’t quite land because the description used “gray” instead of “overcast”.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#415

Post by [deleted] »

simonh wrote: post #237
tempestn wrote: post #230

LLM output, as the author acknowledges here, is already non-deterministic. Next token probabilities are set, and tokens are chosen pseudo-randomly. As I understand it, this watermark is just going to be a matter of using a known seed and algorithm to make those pseudo-random choices, such that a signature can be detected. The important thing is, it's not replacing intentional choices with random ones, it's just gener…

That cannot be true. The quality of an LLM's output is the quality of the probability calculations for the next token. Anything that degrades the relationship between the system's best assessment of the appropriate probability and the actual probability used is a degradation of the quality of that probability and therefore of the output. If this didn't have a detectable effect on the quality of the token probability…

[deleted]

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#416

Post by CuriouslyC »

levocardia wrote: post #26

Crazy how a smart person like this fails to understand the gumbel softmax technique. It does not affect writing quality at all, provably. The very fact that there is generally no "best next token" with 100% certainty is precisely why the trick works (you cannot watermark a response to "respond with the To be or not to be soliloquy from the first folio Hamlet", for precisely this reason).

By definition watermarking narrows and biases the response distribution. Clever algorithms might reduce the perceptual impact and minimize some cherry picked metrics, but it's still worse.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#418

Post by cadamsdotcom »

inigyou wrote: post #115
case540 wrote: post #95

Earlier quoted context omitted.

Citation needed

You know you can just try it and see on any inference system thst has this knob, right? Related: if you don't have a limit on sampling (top-K or top-P), eventually you'll hit one of the really unlikely tokens by chance and then the model will switch to Japanese because the most likely completion after a random Japanese character in the middle of an English sentence is more Japanese writing, not a reversal back to Eng…

That could be how it works, but in practice it takes into account all previous tokens when producing the next-token distribution to sample from. So a switch back is more likely than your explanation supposes.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#419

Post by eloisant »

voidUpdate wrote: post #199

> "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much

Additionally, LLM are already rolling the dice on different wordings so I don't see how this watermarking makes it any less precise.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#420

Post by breezybottom »

dofm wrote: post #336
NitpickLawyer wrote: post #283

Earlier quoted context omitted.

> He is not using AI That's ... even worse? So we're all here in the comments trying to figure out what the author means, and what their overall point is, while clearly they don't even use the damn thing? Oof... What a waste of time for everyone involved.

Why? I really don't understand this. Why can't a tech writer take a deep but neutral intellectual interest in something? Isn't it important that some do? Do people have to be stakeholders or clearly on one given team, pro- or anti-, for their opinion to matter? Is it that tribal? It seems fully logical to me that someone who writes for a living (who, as it happens, developed the very markup language LLMs use for ever…

If he writes then he should have no investment in LLMs. Humans have been able to write for thousands of years.
Post reply on HN