Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

121–130 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#121
post #19

Earlier quoted context omitted.

I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want. On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense. Even good human writers would probably struggle with this constraint. It wou…

It's a bit like trying to finish a sentence when you're really stoned... you vaguely remember the preceding couple of words you've said but don't really know how you got there and now you're wandering in the forest trying to stumble on coherency. Well, I suppose it's nearly the opposite of that experience, upon further review. But for some reason, that's where my head jumped.

After "stoned" you triggered into a different state.

You be a human who's brain shifted into LLM mode (chainneling Markov?).

Or perhaps you're an LLM impersonating humanity.

I often wonder how much LLMs are just mirroring our own brain's patterns.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#123
post #82
post #74

Earlier quoted context omitted.

It didn’t take, apparently.

It’s fully possible I didn’t explain it very well in the first place, but he is making a wider point. The point I made (quite briefly) is that watermarking is only feasible because for good writing it is necessary to use T>0, or the writing will never explore a more creative choice, and that at T=0 you don’t even need a watermark to spot LLM-generated text. The point he is making is consistent with this, isn’t it? Ei…

What does he think of all the other adulterations of LLMs that already happen?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#124
post #117

Translation: No one can ever again use Claude for proofreading their own prose unless they’re willing to risk that the whole thing might be flagged as having been generated by Claude. I think that was intended, yes.

The thing that could change is interpreting "the whole thing as generated by Claude"

Well yeah, if it's output from Claude it's likely to get detected as being output from Claude.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#125
post #31

Earlier quoted context omitted.

As I understand it, the current watermarking methods rely on a secret key, making the detection schemes a black box to anyone not in possession of the key. This means organizations like Anthropic are free to make any claim about authorship they want, true or not, and no one can call them on it.

Part of the legislation requires them to make a public AI text detector (ala GPTZero I assume). Wouldn’t having that be enough to eventually reverse engineer the key?

Probably not to get the key, but you could certainly use it adversarially to remove the watermark.

Removal may come down to changing every third token to a different one.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#126
post #12

Earlier quoted context omitted.

Perhaps LLM outputs are uncopyrightable, but derivative works of copyrighted works are not automatically in the public domain.

That's an intriguing twist, isn't it? It could lead to a tug-of-war. Working backwards: if it is possible to confirm 100% confidence that a chunk of text is LLM output, then it is "PD until proven otherwise". How can a human reliably assert human authorship of their source text? When all watermark tests fail? Is that proof of humanity now? If a human proves human authorship, and LLM watermarking tests positive, then…

How do you assert it now? I post some text on the internet, you claim you have copyright, how do you prove that?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#127
What is even the point of watermarking AI slop supposed to be? All it does is provide people with the false implication that anything which is not watermarked must not be AI-generated.

I struggle to see how this could possibly be useful unless there's some sort of psy-op going on to trick people into uncritically accepting anything lacking a watermark as not being AI-generated.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#128
post #9

> I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo ran…

That's inaccurate in two ways: (1) The behavior that is approximately what you describe is not "fundamental" (though it may not be something you can disable on some hosted providers), it is an option that is not fundamental (and with runtimes where you have full control can be either disabled or tuned in a large number of manners), and (2) The actual behavior that is approximately what you describe already usually in…

> which inherently compromises quality.

I don’t see how this follows? Tokens are chosen randomly. If you choose tokens with a different RNG in the same distribution, you’re still getting equally good or bad tokens.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#129
post #3

> The provider must mandate in their terms-of-service that users not remove the watermarking. So, you don't own the generated text, and can't use it freely then. What if I copy paste a section, or rewrite a section of text to my liking? What if I rewrite some lines of code that contains the mark? Security theater, and vague enough to be used as a weapon against who the government wishes. I hope it's left off for non-…

You know you can just ignore EU laws outside of the EU
Post reply on HN