Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

291–300 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#291

Post by vanviegen »

chii wrote: post #274
ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

> blindly trusting they won't train on any of that being allowed to train on any data that you can legally obtain ought to be a right for anyone. After all, i am allowed to learn off anything i can legally read (and perhaps even illegally read). The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.

> The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.

But LLMs are replacing the original, just in different words.

And what does 'legally obtain' mean in this context? Copyrighted content is usually licensed for specific purposes. So if a license is given from training your LLM, then by all means do! But what if the license is 'for personal use'... ?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#292

Post by sbszllr »

I commented it last time the post about Claude watermarking went viral and I'm going to say the same thing again:

"I've been working in the media and model IP space for quite many years. What this article misses a bit is the threat model for watermarking in general. Watermarking and fingerprinting have inherently weak security guarantees -- they rely a lot on security through obscurity, weak assumed adversaries to deliver. There are clear trade offs between true positives, false positives and maintaining the quality of the media. It's true for audio-visual media, models and their outputs alike. As much as I like to take shots at poor technical choices by corps and govs, this one is unjustified. Sure, inserting glyphs is bad but biased sampling is as good as it gets in 2026."

Since the announcement, there have been many people who don't seem to fully understand what a security guarantee is, what trade offs it might involve, or how popular the type of technological solution is in general (media watermarking is ubiquitous).

And naturally, there are challenges with how you will make sense of the score in your org, e.g. you wrote an email, and it's flagged as LLM-generated because you copied two generated/edited paragraphs.

Yeah, the article might disagree with watermarking as a matter of principle, comparing it to censorship. But the methodological arguments that I have read so far have been thin in these articles.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#293

Post by zrm »

josephg wrote: post #268
ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

I think this is a very real concern. But I’m not sure of any way around it. Any stenographic system that you have the code for can be trivially defeated. I wonder if this would be a good use for homeomorphic encryption. There might be a way to let anthropic check some text without actually giving them access to the source text. Any experts around? We could use your skills!

> Any stenographic system that you have the code for can be trivially defeated.

They're giving you an oracle regardless, which is almost as good. Take LLM output, make some modification, ask the detector if it's LLM output, repeat until you learn what kind of changes you have to make to defeat it.

Or don't even bother learning what to do, just make arbitrary changes until it says it's not, so when the person they're submitting to does the same check it says the same thing.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#295

Post by tjpnz »

thinkingemote wrote: post #247

I think we will see a hidden motivation behind this as ultimately so the output can have an author, the author can be attributed and finally the output will be copyrighted and so the LLM has more value. Everyone who invests in AI companies wants to see the value of their investment increase. I'd give it about 3-5 years until an AI company claims copyright over code their LLM produces. This is a crucial step in that p…

As someone who doesn't utilize LLMs for writing production code I'm looking forward to starting a consultancy and reaping the rewards.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#296

Post by AshleysBrain »

voidUpdate wrote: post #199

> "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much

If the exact words we choose when writing matter so much, then why use a non-deterministic LLM that produces slightly different output on every run?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#297

Post by troupo »

0x_rs wrote: post #101

I'd encourage reading this paper, and literature on scaling laws in autoregressive models: https://arxiv.org/abs/2303.11156 Total variation distance has been measured to decrease as you scale a model, and that is the primary mechanism "watermarking" as discussed in the Anthropic announcement relies on. It becomes more difficult to reliably detect text as a fixed sample count without tweaking the distribution further.…

EU is a capitalist union first and foremost. So their regulations try to straddle the thin line between "regulate to death and tell companies exactly how to do things" and "the companies can do whatever the hell they want".

Since most Americans are in the latter camp, anything that even hints at making companies responsible for anything is viewed as being squarely the former.

Whereas most EU regs are "play nice, be responsible, behave like adults. If not, this can always turn ugly". Same here.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#298

Post by Chrisszz »

This is the yet another embarrassing idea from the EU they came up with, if they could focus less on dumb things and more on providing real support to the development of technology maybe we could have more healthy competition that will eventually lead to better overall technology in the hands of everyone instead of acting like dumb chickens and not just being passively useless but also stopping the real labs from doing the real work into doing this bs

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#299

Post by troupo »

krackers wrote: post #13
andy99 wrote: post #8

Earlier quoted context omitted.

I should have read that, it’s actually quite reasonable and I don’t really understand the objections in TFA having read it. > One of my fundamental problem with this is that no two synonyms carry the exact same meaning. “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter.…

I don't understand Gruber's points either, I wonder if there is some fundamental technical misunderstanding. Does he think that the logits should be sampled from in a "pure" manner without introducing any other bias? Does he know that there's already a sampling temperature, and that most providers have probably moved on to sampling strategies other than top-k? Does he know that the word choices have already been alte…

> I don't understand Gruber's points either,

Gruber's point is to bash the EU. He couldn't care much about anything else.

Post reply on HN