Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

311–320 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#311

Post by exe34 »

chii wrote: post #274
ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

> blindly trusting they won't train on any of that being allowed to train on any data that you can legally obtain ought to be a right for anyone. After all, i am allowed to learn off anything i can legally read (and perhaps even illegally read). The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.

Oh so if I use mickey mouse in a completely original production that doesn't replace the existing work by Walt Disney, you reckon they'll be fine with that?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#312

Post by KaiserPro »

smallerize wrote: post #7

Translation: No one can ever again use Claude for proofreading their own prose unless they’re willing to risk that the whole thing might be flagged as having been generated by Claude. I think that was intended, yes.

wait what?

but proof reading is a linter, not a writer. the proof reader will say "I think this is clumsy can you try x,y & z"

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#313

Post by gmerc »

TeMPOraL wrote: post #304
bonzini wrote: post #282

Earlier quoted context omitted.

It's a bit different when "training on any data" means basically storing a lossily-compressed copy of that data, that could be spit out years later if the model decides to do so.

It's exactly the same problem as with humans, though. It's part of why we sign NDAs, and why their duration is measured in years (and that's not even targeting the human retention - just duration after which information ages enough that its disclosure is not likely to negatively impact anyone who cares).

It’s not because humans are rate limited which is the entire point.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#314

Post by passwordoops »

chii wrote: post #274
ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

> blindly trusting they won't train on any of that being allowed to train on any data that you can legally obtain ought to be a right for anyone. After all, i am allowed to learn off anything i can legally read (and perhaps even illegally read). The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.

You are one person. The corporation is not. Scale matters

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#315

Post by Alpha3031 »

GaryBluto wrote: post #302

I wonder if this watermarking system to lead to an increase in tortured phrases, believed to be caused by plagiarism evasion tools that change random words to applicable synonyms. https://arxiv.org/abs/2107.06751

Seems unlikely unless Anthropic invented a time machine, given that the phenomenon predated Claude 1 by two years, and their stated introduction of watermarking (August 2) by 5.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#316

Post by rimliu »

amanzi wrote: post #88

I was initially surprised that Gruber was so invested in the "quality" of AI-generated text, which in my mind is an oxymoron. But really, Gruber's interest here is with the EU. This forms part of his ongoing attacks on the EU, all because they have been forcing Apple to align with regulations.

I am from EU. Alas it has a tendency to produce some idiotic regulations. Cookie banner, new packaging fee, etc. I genuinely think some Apple related ones hurt customers more than help them.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#317

Post by TeMPOraL »

tempestn wrote: post #232
dragonwriter wrote: post #15

Earlier quoted context omitted.

That's inaccurate in two ways: (1) The behavior that is approximately what you describe is not "fundamental" (though it may not be something you can disable on some hosted providers), it is an option that is not fundamental (and with runtimes where you have full control can be either disabled or tuned in a large number of manners), and (2) The actual behavior that is approximately what you describe already usually in…

You're generating a pseudo random number one way instead of another way. How would that inherently compromise quality?

Same reason that you don't just replace your rand() implementation with "return 4; // chosen by fair dice roll". If you need randomness for whatever reason, biasing the generator is compromising quality.

( https://xkcd.com/221/ )

In case of LLMs, you can look at it from high and low level.

At low level - if you could do with less randomness, you can always lower temperature. You usually keep it (or for SOTA providers' chat UI, they keep it) at a level where it's about right level - high enough to allow for more creative leaps and interpretations, low enough that it doesn't go off into crazy land after the third paragraph.

At high level - creativity is driven by randomness. If you had an author (fiction or nonfiction) you like for their both broad and deep range of insightful thoughts, would you be happy if they suddenly developed an acute porn obsession and uncontrollably added lewd subtext to every other sentence? Still creative, still deep, but now with that one strong attractor that biases their every thought in a single direction? Would you trust/enjoy their output as much as you did before?

That, slightly exaggerating to make it more obvious, is what "loss of quality" means here.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#318

Post by simonh »

simiones wrote: post #245
simonh wrote: post #239

Earlier quoted context omitted.

The quality of the LLM just _is_ the quality of the token probabilities it generates. Better quality token probabilities, better quality output. Worse quality token probabilities, worse quality output. Watermarking changes the probability calculations for reasons other than quality. It can't not compromise quality. It literally leads the LLM to occasionally chose different tokens just for watermarking purposes.

No one uses a pure random function over the whole probability distribution described by the LLM's output. For example, there is exactly 0 probability that the chosen next token by any common API or even local LLM runner would be a token whose final value is "0.0001" if there exist at least K tokens whose value exceeds "0.7". Also, as long as the same sampling strategy is used during training as the one used during in…

>No one uses a pure random function over the whole probability distribution described by the LLM's output.

So what? By definition with this system the LLM will chose tokens it otherwise would not, purely for watermarking reasons. Yes this token may have had a decent likelihood of being chosen anyway, but it wouldn't have been chosen and now it was for reasons nothing to do with output quality.

I'm not sure what your last paragraph is trying to say. The blue/green list system changes what output the LLM would otherwise produce. You can't train it to produce watermarked output with this system. If you tried to, there would be no delta between trained output and watermarked output for you to be able to detect.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#319

Post by GaryBluto »

Alpha3031 wrote: post #315
GaryBluto wrote: post #302

I wonder if this watermarking system to lead to an increase in tortured phrases, believed to be caused by plagiarism evasion tools that change random words to applicable synonyms. https://arxiv.org/abs/2107.06751

Seems unlikely unless Anthropic invented a time machine, given that the phenomenon predated Claude 1 by two years, and their stated introduction of watermarking (August 2) by 5.

I see how that was confusingly written. I'm not suggesting Anthropic are somehow retroactively causing it, just wondering if it could create a similar effect.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#320

Post by [deleted] »

ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

[deleted]
Post reply on HN