Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

341–350 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#341

Post by lyu07282 »

TheOtherHobbes wrote: post #227
beering wrote: post #128

Earlier quoted context omitted.

> which inherently compromises quality. I don’t see how this follows? Tokens are chosen randomly. If you choose tokens with a different RNG in the same distribution, you’re still getting equally good or bad tokens.

Not all values of "equally good" are equally good. Writing has rhythm, or at least it's supposed to, and synonym swapping compromises it. Never mind metaphors and similes, which are even more tightly constrained. LLM writing is still a long way from good. Sometimes you get lucky with the odd line, but there's a difference in quality between influencer slop, genre fiction, and literary fiction and/or best-in-class jou…

wow even on HN people have no clue how any of that works? all LLM generation has some inherent randomness to it, if you replace part of that randomness to be deterministically random the result of the generation with the fingerprint and without it, is INDISTINGUISHABLE. This has absolutely nothing to do with "synonym swapping". Its also again people not understanding how anything works missing the real concern, which is that nobody can ever tell if it isn't also secretly a fingerprint with the user and session id.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#343

Post by Create »

"possibly build tools to take a chunk of writing and try to remove your fingerprint from it, i.e, protect anonymity,"

The standard procedure to do this, is to chain translations to other languages and back. The message remains, but the wording will pick up some noise. --Dec 30, 2008.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#345

Post by sebastiennight »

zrm wrote: post #293
josephg wrote: post #268

Earlier quoted context omitted.

I think this is a very real concern. But I’m not sure of any way around it. Any stenographic system that you have the code for can be trivially defeated. I wonder if this would be a good use for homeomorphic encryption. There might be a way to let anthropic check some text without actually giving them access to the source text. Any experts around? We could use your skills!

> Any stenographic system that you have the code for can be trivially defeated. They're giving you an oracle regardless, which is almost as good. Take LLM output, make some modification, ask the detector if it's LLM output, repeat until you learn what kind of changes you have to make to defeat it. Or don't even bother learning what to do, just make arbitrary changes until it says it's not, so when the person they're…

> it says the same thing

Reference needed? I think it remains to be proven whether those detectors can be considered deterministic.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#346

Post by throwthrowuknow »

KaiserPro wrote: post #273
voidUpdate wrote: post #199

> "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much

I mean this is the thing that really comes off hard. If you want precision and clarity of your writing, then you need to hand write it. Just like when you are optimising, its common to drop to a lower level language because the compiler doesn't express what you want. Sure its hard, but you know, thats kinda the point. Even if you don't want that, the LLM is an average of the style it was trained to give out. Which is…

The bigger problem is its effect on the way we use language. The amount of LLM generated or edited media is going to keep increasing and the media that people consume affects their own word choice.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#347

Post by saidnooneever »

ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

its lovely training data. no detection? add to training set -_-.

its also kind of laughable that somehow people are trying to prevent the outputs not to be altered. Asif you cannot manually paraphrase anything you can read. So the only solution would be, to make it utterly unreadable (which is not possible, it obviously defeats the purpose of the thing).

Not to mention local models ofcourse :-)

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#348

Post by its-summertime »

> My error was believing Anthropic that their system wouldn’t adulterate and corrupt the semantics of the text their models generate.

Has that ever been the case? Are they not actively tweaking their models, their fine tuning, the system prompts, the tool definitions and implementations, the guard rails, tool calls, instant responses. There are hundreds of knobs that they can change daily, or between each prompt, or even half way through a generation.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#349

Post by tosh »

any watermarking ai researchers who can explain this?

what if the llm should

- repeat something verbatim (important in a compaction prompt)

- there is just one correct order of tokens for a somewhat long chain (a certain sequence of control signals)

- provide a diff of 2 inputs

without punctuation or whitespace wiggle room?

how does the drifting work?

does it postpone the drifting and drift stronger later?

what if max_tokens is set to a low number?

in what way does this not affect output quality?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#350

Post by dofm »

Imnimo wrote: post #172
dofm wrote: post #82

Earlier quoted context omitted.

It’s fully possible I didn’t explain it very well in the first place, but he is making a wider point. The point I made (quite briefly) is that watermarking is only feasible because for good writing it is necessary to use T>0, or the writing will never explore a more creative choice, and that at T=0 you don’t even need a watermark to spot LLM-generated text. The point he is making is consistent with this, isn’t it? Ei…

>Either you allow temperature to drive creativity, consistently in a way that can be influenced and analysed, or you adulterate that process for the purposes of meeting a corporate/legal directive, in a way that is proprietary and obscure This framing does not make sense to me. What do you mean by "influenced and analysed"? How have you or anyone been influencing or analyzing the randomness behind the sampling proces…

> How have you or anyone been influencing or analyzing the randomness behind the sampling process to create better writing?

You're mischaracterising or misunderstanding my point, or I mangled it.

I mean it is possible to analyse, control, monitor, study the impact of changing temperature on the writing, yes?

The point about watermarking is that this relationship — change the temperature, see the effect — is now being adjusted by an unstated, secret process you explicitly can't control.

(I gather Anthropic have recently taken away this setting anyway; that was news to me.)

Post reply on HN