Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

231–240 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#231

Post by mrweasel »

This seems like a non-issue, or maybe I have the wrong expectations about writing. You write a text, ask Claude to proof-read it, but then you wholesale just copy Claudes output and use that as the final text? Wouldn't you review the changes it suggests and only take those you agree with, there by completely bypassing the watermarking?

Alternatively, you ask Claude to write the whole thing and proof read it yourself. In that case I'd like to know how much you'd need to change to break the watermarking, i.e. how much of a text would you need to change for it to be considered your work and not that of Claude?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#232

Post by tempestn »

dragonwriter wrote: post #15
syrrim wrote: post #9

> I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo ran…

That's inaccurate in two ways: (1) The behavior that is approximately what you describe is not "fundamental" (though it may not be something you can disable on some hosted providers), it is an option that is not fundamental (and with runtimes where you have full control can be either disabled or tuned in a large number of manners), and (2) The actual behavior that is approximately what you describe already usually in…

You're generating a pseudo random number one way instead of another way. How would that inherently compromise quality?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#233

Post by Kwpolska »

wpietri wrote: post #78
levocardia wrote: post #26

Crazy how a smart person like this fails to understand the gumbel softmax technique. It does not affect writing quality at all, provably. The very fact that there is generally no "best next token" with 100% certainty is precisely why the trick works (you cannot watermark a response to "respond with the To be or not to be soliloquy from the first folio Hamlet", for precisely this reason).

It seems to me like he started out mad and looked to justify it. I'm skeptical that anybody generating LLM text is really all that concerned about optimal word choice. Or even particularly good prose. But let's pretend that person exists. If that person tried, say, an open model and that same model with watermarking applied, I'd be eager to hear their thoughts on the prose quality. Especially if they built an experim…

Considering Gruber's always comically butthurt about regulation, especially EU regulation, your theory seems accurate.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#234

Post by croon »

shubhamjain wrote: post #221
voidUpdate wrote: post #199

> "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much

What an absurd take on a very reasonable concern. As much as I hate “obviously AI” writing, I don’t understand why we have to handicap the tech and prevent it from improving. This manipulation of word choices virtually guarantees AI will always sound like a robot.

I don't think there's been a single version of the major LLM providers that haven't handicapped the tech since the beginning by changing temperature/top-P/frequency penalty/etc. All of those stray from "highest probability" token selection. What's funny is it was done specifically to make it sound less like a robot/deterministic.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#235

Post by georgemcbay »

kgarten wrote: post #210
voidUpdate wrote: post #199

> "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much

One could also use butterflies to write ;) https://xkcd.com/378/ The problem is, that LLMs worked very well for me to improve my writing. Especially as I'm not a native speaker, it was a great way to improve the legibility of my work. I want a tool that helps me improve my writing. A tool I can learn from. Not a tool that switches out "bananas" to "airplanes." I'm was using Claude Opus and now Fabel extensively for e…

> improve the legibility of my work.

Nitpicking here in a way that I would usually avoid, but it is relevant to the conversation being had and it seems like you might appreciate the information... "Readability" would be the more correct word to use here instead of "legibility".

Legibility is close enough for me to know what you mean based on the context, but it really applies to the visual presentation and how easy something is to read at a symbolic level (whether someone's handwriting or font choice is good or bad impacts legibility, whether someone uses good grammar or not impacts readability).

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#236

Post by tempestn »

TheOtherHobbes wrote: post #227
beering wrote: post #128

Earlier quoted context omitted.

> which inherently compromises quality. I don’t see how this follows? Tokens are chosen randomly. If you choose tokens with a different RNG in the same distribution, you’re still getting equally good or bad tokens.

Not all values of "equally good" are equally good. Writing has rhythm, or at least it's supposed to, and synonym swapping compromises it. Never mind metaphors and similes, which are even more tightly constrained. LLM writing is still a long way from good. Sometimes you get lucky with the odd line, but there's a difference in quality between influencer slop, genre fiction, and literary fiction and/or best-in-class jou…

It's not just swapping synonyms. The way llms work is by predicting the likelihood of the next token. It's inherently probabilistic. Choices are made based on weighted random number generation, based on those probabilities. Changing how you generate the random numbers doesn't degrade the output.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#237

Post by simonh »

tempestn wrote: post #230

LLM output, as the author acknowledges here, is already non-deterministic. Next token probabilities are set, and tokens are chosen pseudo-randomly. As I understand it, this watermark is just going to be a matter of using a known seed and algorithm to make those pseudo-random choices, such that a signature can be detected. The important thing is, it's not replacing intentional choices with random ones, it's just gener…

That cannot be true. The quality of an LLM's output is the quality of the probability calculations for the next token. Anything that degrades the relationship between the system's best assessment of the appropriate probability and the actual probability used is a degradation of the quality of that probability and therefore of the output.

If this didn't have a detectable effect on the quality of the token probability calculation, the watermark wouldn't be detectable. It may be a small degradation in the quality of the output relative to the neds of many users in many situations, but it's not zero. It's literally sometimes choosing different words that it otherwise would specifically for watermarking purposes.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#238

Post by TheOtherHobbes »

The issue here isn't (just) adulteration, it's that watermarking in general is unworkable.

If all the providers use watermarking systems with different shifting logit weightings, and the keys are secret, you have to check every provider to see if it produced a given text.

Which is clearly ridiculous.

And if all providers collaborate and use the same weightings, or if the weightings are constant and not rotated cryptographically, a generic watermark remover becomes trivial.

That's not even getting into the legal complexities of businesses running open source models without watermarking locally.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#239

Post by simonh »

syrrim wrote: post #9

> I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo ran…

The quality of the LLM just _is_ the quality of the token probabilities it generates. Better quality token probabilities, better quality output. Worse quality token probabilities, worse quality output.

Watermarking changes the probability calculations for reasons other than quality. It can't not compromise quality. It literally leads the LLM to occasionally chose different tokens just for watermarking purposes.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#240

Post by woadwarrior01 »

I think writing is the killer use case for local LLMs.

We've had so many advancements in LLM samplers for improved text generation (off the top of my head: min-P, adaptive-P, XTC, DRY, p-less, Top-H, Top-n-Sigma, and so many more) but hosted LLM APIs only provide three basic knobs: temperature, top-k and top-p which are old as the mountains in LLM years at this point.

One thing that doesn't help the local LLM case is that all the popular VC backed local LLM wrappers also only support the same three ancient knobs because I suppose they're more preoccupied with their next fundraise than with keeping up with the advances in tech.

Post reply on HN