Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

111–120 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#111

Post by inigyou »

npilk wrote: post #90
hyusap wrote: post #36

Earlier quoted context omitted.

autoregressive generation doesn’t mean the model is myopic. the next-token distribution can already reflect a longer horizon plan for the output sequence.

Sure, but mightn’t there be several plausible long horizon plans? Here’s an example: I had asked Claude for some music recommendations in a certain style. Part of its output was: — *Long journey tracks* Clinic — “The Return of Evil Bill” Guided by Voices — not really, wrong band Silver Apples — “Oscillations”. Proto-everything, deeply repetitive, hypnotic. — So at some point there, the next token produced was “Guided…

Would be fun to run an LLM on fake output from itself. Like just force the first N tokens to say the beginning of something really stupid, and then see how it finishes the sentence. "You're absolutely right! Human feces is actually the most effective engine coolant because $"

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#112

Post by moralestapia »

inigyou wrote: post #109
moralestapia wrote: post #106

Earlier quoted context omitted.

There are diffusion-based models and transformer-based models (and many other "architectures"), so your comment does not make sense.

Are there any diffusion-based or otherwise non-transformer-based models in mainstream use?

If by mainstream you mean "ChatGPT" or "Claude", then no.

But also, no one really knows as they're closed.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#113

Post by inigyou »

hyusap wrote: post #36
npilk wrote: post #19

Earlier quoted context omitted.

I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want. On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense. Even good human writers would probably struggle with this constraint. It wou…

autoregressive generation doesn’t mean the model is myopic. the next-token distribution can already reflect a longer horizon plan for the output sequence.

It can but it is limited because it's only got a single pass through the network to fit the entire "longer horizon plan".

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#114

Post by demibabs »

pessimizer wrote: post #96
ButlerianJihad wrote: post #22

Earlier quoted context omitted.

That's an intriguing twist, isn't it? It could lead to a tug-of-war. Working backwards: if it is possible to confirm 100% confidence that a chunk of text is LLM output, then it is "PD until proven otherwise". How can a human reliably assert human authorship of their source text? When all watermark tests fail? Is that proof of humanity now? If a human proves human authorship, and LLM watermarking tests positive, then…

The watermark, if I'm understanding correctly, only proves that something has been touched at some point by a particular model, not that it was entirely generated by a particular model. As for it being "uncopyrightable" if it were the output of an LLM, I think computer people are making a very aspie interpretation of a single decision. I think it's more that the LLM (and thereby its owners) cannot itself hold a copyr…

> The watermark, if I'm understanding correctly, only proves that something has been touched at some point by a particular model, not that it was entirely generated by a particular model.

For the watermark to be detectable, the text needs to be like 75% AI generated.

If you have an LLM “touch” one section of the article, it’s not gonna be detectable.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#115

Post by inigyou »

case540 wrote: post #95
reliablereason wrote: post #28

Earlier quoted context omitted.

(1) LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time. This is a consequence of how they are trained.

Citation needed

You know you can just try it and see on any inference system thst has this knob, right?

Related: if you don't have a limit on sampling (top-K or top-P), eventually you'll hit one of the really unlikely tokens by chance and then the model will switch to Japanese because the most likely completion after a random Japanese character in the middle of an English sentence is more Japanese writing, not a reversal back to English.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#116

Post by dare944 »

demibabs wrote: post #107
dare944 wrote: post #31

Earlier quoted context omitted.

As I understand it, the current watermarking methods rely on a secret key, making the detection schemes a black box to anyone not in possession of the key. This means organizations like Anthropic are free to make any claim about authorship they want, true or not, and no one can call them on it.

Part of the legislation requires them to make a public AI text detector (ala GPTZero I assume). Wouldn’t having that be enough to eventually reverse engineer the key?

Not if they designed the algorithm right.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#117

Post by smb06 »

smallerize wrote: post #7

Translation: No one can ever again use Claude for proofreading their own prose unless they’re willing to risk that the whole thing might be flagged as having been generated by Claude. I think that was intended, yes.

The thing that could change is interpreting "the whole thing as generated by Claude"

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#118

Post by inigyou »

avaer wrote: post #49
syrrim wrote: post #9

> I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo ran…

That's missing the point. It's the distribution that's the "best", not the tokens. Then Anthropic comes in and makes the distribution something other than the best. The only saving grace is that Anthropic says it's not that bad. Even so, I don't think it will stop here. Once this is in place, the next step is to put more and more identification into the AI generated content; might as well pack it in, it's not that ba…

there's absolutely no reason to think Claude produces absolute best token distributions or that slight adjustments would be noticeable.

In fact we know it's not that good because we can often tell Claude's writing apart from human writing.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#119

Post by Maxatar »

brookst wrote: post #68
npilk wrote: post #19

Earlier quoted context omitted.

I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want. On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense. Even good human writers would probably struggle with this constraint. It wou…

This is the classic misunderstanding that LLMs only pick the next token at a time. Really, they are coalescing the probabilities of a range of tokens at a time. There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”.

>There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”.

You're mixing up two claims here, and only one of these is kind of true. Yes LLMs do internally plan ahead in a way that is emergent rather than strictly part of their architecture, so that part of your claim is true. The way you word it by saying they are "coalescing the probabilities of a range of tokens at a time" is poetic sounding jibberish though. What's actually happening is one distribution output for the next token computed from a hidden state that implicitly encodes where the text headed.

Your claim that if an LLM does happen to pick a token "th" instead of "tw", then the LLM isn't stuck with that decision is entirely false for autoregressive LLMs which is what all of the frontier models are. Whatever an LLM picks as its output token is final, it has no ability to undo that token selection and it must continue on the basis of that choice. It can't go back on that decision and revise the output.

If you're interested in this, Anthropic has a summary of a very technical paper on this topic that mostly deals with this issue with respect to poetry:

https://www.anthropic.com/research/natural-language-autoenco...

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#120

Post by inigyou »

beering wrote: post #70
avaer wrote: post #49

Earlier quoted context omitted.

That's missing the point. It's the distribution that's the "best", not the tokens. Then Anthropic comes in and makes the distribution something other than the best. The only saving grace is that Anthropic says it's not that bad. Even so, I don't think it will stop here. Once this is in place, the next step is to put more and more identification into the AI generated content; might as well pack it in, it's not that ba…

The watermark doesn’t change the distribution, only per-token selection. I think not understanding that is the source of most people’s FUD.

There's no difference between those two things. The distribution that matters is the distribution of tokens that are picked not the distribution of tokens the LLM model passed to the selector.
Post reply on HN