Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

511–520 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#511

Post by lanternfish »

Phemist wrote: post #506
tempestn wrote: post #232

Earlier quoted context omitted.

You're generating a pseudo random number one way instead of another way. How would that inherently compromise quality?

1. As watermarked text is added to the training data, watermark-related tokens will be associated more with AI outputs and thus lower quality outputs which will hasten model collapse. Especially because every provider has its own secret key and they are all training on eachother's outputs anyway. I guess they can at scale filter the watermarked documents (by necessarily allowing eachother to at scale checked for wate…

There are no watermark related tokens, there is no watermarked set - read the paper, it's public and not that complicated. It doesn't change the distribution of completions, and won't change the situation where one output token has the majority of probability mass.

You're missing the fundamentals here!

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#512

Post by elim_garak »

mangoman wrote: post #428

I’m surprised by the comments here being so favorable to anthropic. The comments are right about there being no “best” token, and yeah Gruber may have an agenda here. But I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. Take the “gray” and “overcast” choices. And lets say before applying synthid the percentages were 48% and 52%. Those pe…

> To change those percentages to 45% and 55% in a non learned way makes it seem like training wasn’t important?

I think (or I hope, anyway) that this overestimates how much impact the tweaks actually have on text.

The model has some things it "wants" to say. If it wants to tell a story about how someone reacted to dreary weather, it's going to tell approximately the same story regardless of whether the dice-roll caused it to describe the weather as "gray" or "overcast". And because "gray" and "overcast" were _already_ possibilities, the tweak from 52% -> 55% is completely lost in the noise.

But it's true that this is all based on hope. I'm confident that you could make the tweak against arbitrary prose and even a true artiste like Gruber would never be able to tell the difference. I'm less confident that there isn't some edge case somewhere that causes a tweak to be worse than 3%, especially in some narrow application where word choice _does_ matter (like law). Even then, though, laws are already written by people who are as noisy if not noisier than LLMs.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#514

Post by kergonath »

A lot of that rant is nonsense. Is Gruber also outraged that a RNG is already involved in everything LLM produce with a non-zero temperature? Surely it already leads to widespread use of “non-optimal words”. Simply the idea that there is a single optimal choice, down to every single word, to convey a meaning and anything straying from that is adulteration is laughable. It’s very difficult to take seriously an argument based on this kind of foundation.

He had a very similar rant a couple of days ago when he somehow thought that they would use invisible characters. It’s just as useless.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#515

Post by redfloatplane »

jorisw wrote: post #453
redfloatplane wrote: post #286

Earlier quoted context omitted.

Yes, I decided to stop reading his blog relatively recently after some extremely hot takes on EU policy. I don't feel his thoughts on the matter are particularly well-thought-out, and I feel like he's just stanning for Apple from his priors rather than from any grounding in reality. I dunno, I guess that's what you should expect from Gruber but these EU-bashing articles lowered the enjoyment I got from his blog under…

As an EU citizen I've found myself in agreement with everything he's written regarding EU policy

Good for you! As an EU citizen I have found myself in almost total disagreement with everything he's written regarding EU policy. Horses for courses.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#516

Post by Helloworldboy »

summarybot wrote: post #505
nemo wrote: post #376

Earlier quoted context omitted.

>A lot of people use Claude as a friend/therapist/romantic partner People developing a para-social (pseudo-social?) relationship with a corporate robot have far bigger problems than the word-chooser in their robot "friend".

Very intelligent people need intelligent-others to bounce ideas off of, and the LLM can be that.

[dead]

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#517

Post by Imnimo »

dofm wrote: post #350
Imnimo wrote: post #172

Earlier quoted context omitted.

>Either you allow temperature to drive creativity, consistently in a way that can be influenced and analysed, or you adulterate that process for the purposes of meeting a corporate/legal directive, in a way that is proprietary and obscure This framing does not make sense to me. What do you mean by "influenced and analysed"? How have you or anyone been influencing or analyzing the randomness behind the sampling proces…

> How have you or anyone been influencing or analyzing the randomness behind the sampling process to create better writing? You're mischaracterising or misunderstanding my point, or I mangled it. I mean it is possible to analyse, control, monitor, study the impact of changing temperature on the writing, yes? The point about watermarking is that this relationship — change the temperature, see the effect — is now being…

I don't think watermarking breaks this relationship. Watermarked text is still being sampled from the model's output distribution, and adjusting the temperature still has the same affect on that output distribution.

I think a good intuition here is that watermarking is sort of like picking a specific PRNG seed. It's not changing or interfering with the temperature - we're still sampling from the model's probability distribution. But we're making it so the analog of the PRNG seed is coupled to the previous context.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#519

Post by mrguyorama »

shubhamjain wrote: post #221
voidUpdate wrote: post #199

> "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much

What an absurd take on a very reasonable concern. As much as I hate “obviously AI” writing, I don’t understand why we have to handicap the tech and prevent it from improving. This manipulation of word choices virtually guarantees AI will always sound like a robot.

The entire watermarking scheme is based on replacing a random number generator with a seeded random number generator.

This cannot change the "voice" of the LLM. It was already letting a random number generator choose which adjectives to use. Now that random number generator encodes a tiny signal.

But fundamentally the way it writes has not changed.

It's not making it choose different words. It's a minor change to how it chooses between multiple nearly identical words, where in the current case it literally flips a coin.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#520

Post by kergonath »

recursive-call wrote: post #367
voidUpdate wrote: post #199

> "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much

The exact words matter to people who people who don’t use it to write for them. A lot of people use Claude as a friend/therapist/romantic partner/etc. That’s who I imagine would be most affected

> The exact words matter to people who people who don’t use it to write for them.

Right. If the exact words matter, using a non-deterministic LLM is a terrible idea in the first place. I hope these people never try putting the same prompt in different sessions.

Also, now I am curious. How would these people interact with other humans? Is there anyone on earth who would provide the exact same reply, down to every single word, if we asked them the same question more than once?

Post reply on HN