Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

361–370 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#361

Post by unbelievr »

ThePhysicist wrote: post #338
ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

Would really like to know how their watermarking technique works, and if they can use it to store arbitrary information in the text (I assume they can if only in a limited way). I assume if the text is long enough they could add all kinds of metadata that would then be undetectable as the model that generated the text is the "cryptographic" key that encodes the data. I wonder if you can have another model run over th…

The paper for it is open. The technique isn't really hiding information in the text itself, but by forcing some of the rolls to follow a specific pattern. LLMs work by estimating the most likely next token, so there's sometimes a list of possible candidates that would all work in the text (e.g. synonyms). At low "temperature", the output is a bit more deterministic and otherwise it's a weighted dice roll of which token/word to pick. An LLM can loop over existing text and figure out if the output matches something it would do, similar to checking chess moves against the best computed move for detecting cheating. But the LLM purposefully creates a pattern of alternating weighted rolls that are highly unlikely to appear in normal text, and that becomes the watermarking.

The upside is that this has very low false positive detection rate, but the downsides are many. It only works on longer pieces of text. The system is fragile, and small edits (or rewrites by a local model) can fool the detection. Only the owner of the model is able to re-run inference at this level, so data must be sent to them for evaluation. And sometimes the token output is basically 100% deterministic because the input asks for the straight answer to a fact, or to recite a quote verbatim. That leaves no room for watermarking at all, unless the model is able to lie.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#362

Post by piaste »

Topfi wrote: post #310
ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

Why not switch it around? Solves problems of privacy (no data upload), is far more doable and it's far more important to be able to verify content has not been tampered with after originating from a human or capture device (like a camera). We also have decades of cryptographic experience in reliably signing text, images, video, etc. and it doesn't break apart because a text is too short. Having proof that content (es…

> Why not switch it around?

Because a malicious human will gladly copy/paste LLM text and sign it with his "I, a human, definitely wrote this academic paper" key?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#363

Post by avadodin »

This is the obvious solution to watermarking text in a way that isn't easily removed by a dumb tool.

The output quality will likely suffer as stated in the article although this can be mitigated to an extent by only enabling it on more irrelevant filler text while leaving the more functional sections untouched.

The solution using Unicode tricks amounts to malicious compliance as only the most unsophisticated users are going to fail to remove the AI watermarks when trying to pass off AI slop as their own prose.

The real solution here is not having stupid EU–tier laws in the first place.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#364

Post by intrasight »

Topfi wrote: post #310
ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

Why not switch it around? Solves problems of privacy (no data upload), is far more doable and it's far more important to be able to verify content has not been tampered with after originating from a human or capture device (like a camera). We also have decades of cryptographic experience in reliably signing text, images, video, etc. and it doesn't break apart because a text is too short. Having proof that content (es…

Thank you for stating clearly situation. I fully agree with your assessment. For almost 10 years now I have been saying that we need to virtually watermark reality. By "virtual" I mean store the metadata about the digital capture on a public blockchain. Then my devices could have a built-in "fake vs real" detector.

Artists, photographers, journalist, etc. are going to want and need this.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#365

Post by ShinyLeftPad »

ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

That's why detectors like Pangram exist too I think.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#366

Post by epsteingpt »

The idea that language models can write anything remotely useful yet is - mostly - a fallacy.

It's a good idea for many human endeavors to be able to identify AI writing. Communication, after all, is our main way of building the social fabric.

However - and crucially - good writing is still beyond the frontier of any model I've seen so far.

Watermarks for the things that truly matter may not be important at all.

Finally, as X commentators have shown, simply removing punctuation or changing a word here or adding an adverb there manually will screw up the whole process enormously.

The best will be the clever folks who retroactively apply the model distribution to fraud or other crimes to try to implicate the companies via watermark.

Gotta feel for their product, policy and legal team.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#367

Post by recursive-call »

voidUpdate wrote: post #199

> "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much

The exact words matter to people who people who don’t use it to write for them. A lot of people use Claude as a friend/therapist/romantic partner/etc. That’s who I imagine would be most affected

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#368

Post by josephg »

piker wrote: post #271
josephg wrote: post #268

Earlier quoted context omitted.

I think this is a very real concern. But I’m not sure of any way around it. Any stenographic system that you have the code for can be trivially defeated. I wonder if this would be a good use for homeomorphic encryption. There might be a way to let anthropic check some text without actually giving them access to the source text. Any experts around? We could use your skills!

Won't we just be able to fine tune OSS models to detect these patterns across providers? It will be cat-and-mouse but my bet is it converges to a central detector that isn't affiliated with any model provider.

> Won't we just be able to fine tune OSS models to detect these patterns across providers?

A good fingerprint should make use of cryptographic signatures. Without knowing the keys, the fingerprint should be indistinguishable from noise (or just random token selection)

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#369

Post by balherian »

A) without water marking, llm will poison the open internet and poison their own source of fresh new material. B) without watermarking, it potentially exposes an infinite deluge of garbage getting to people , fostering hate, eventually leading to dis-engagement, ergo destroying their training pipeline. C) ...Sadly, them water marking the claude chats with absolute dementia levels of output text is beyond me, i seriously can't tolerate this bullshit "randomly replace words with "similar" words", bs anymore, the output that claude makes right now is outright *corrosive* to my eyes, eventually leading to their training pipeline dying if devs dis-engaging ( of the 3 i think this one is the worst )

I think they need to get their shit-together and realize this is a death warrant for the tech ( in my opinion ).

Post reply on HN