Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

301–310 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#301
post #274
post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

> blindly trusting they won't train on any of that being allowed to train on any data that you can legally obtain ought to be a right for anyone. After all, i am allowed to learn off anything i can legally read (and perhaps even illegally read). The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.

At the risk of stating the obvious, there are a lot of legal rights that are human-specific (voting, holding office, filling lawsuits, etc.). It's not at all obvious why you think that you as a human being legally allowed to learn from something implies that it should be legal to train an LLM on.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#303

Gruber shows here that he really doesn’t understand the basics of how LLM text generation works. It’s weird he picked this battle about the quality of writing in LLMs. Was he planning to use LLMs to write his articles? Well, not that weird actually. He just has a hard-on against anything that comes from the EU since Apple got in trouble. If the EU said tomorrow that they want peace in the world he’d be in Fox News th…

Gruber went from the naively wrong claim that it would insert secret hidden characters (which would be trivial to remove, obviously), to quickly writing a giant essay as if he's an expert on LLMs. Like you said, he is strangely fixated on the EU, and is certain any EU rule is the worst thing in the universe, and this whole piece seems motivated by that guiding force.

Further he later compares Gemini to Anthropic models, saying the latter "writes better", emptily ascribing this to the synthid stuff. I think he heard that Anthropic currently has superior models, but it certainly isn't because they "write better", and if anything Opus 5 now is virtually unintelligible, before the fingerprinting.

The fingerprinting stuff sounds weird. If the EU wants it, it should be limited to the EU, and Anthropic is fully capable of doing that but clearly saw value in recognizing their own output. Is it going to destroy the quality of the output? We'll have to see, and this anti-EU piece, predicated on utter ignorance of the field, is not convincing.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#304
post #274

Earlier quoted context omitted.

> blindly trusting they won't train on any of that being allowed to train on any data that you can legally obtain ought to be a right for anyone. After all, i am allowed to learn off anything i can legally read (and perhaps even illegally read). The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.

It's a bit different when "training on any data" means basically storing a lossily-compressed copy of that data, that could be spit out years later if the model decides to do so.

It's exactly the same problem as with humans, though.

It's part of why we sign NDAs, and why their duration is measured in years (and that's not even targeting the human retention - just duration after which information ages enough that its disclosure is not likely to negatively impact anyone who cares).

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#305
post #274
post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

> blindly trusting they won't train on any of that being allowed to train on any data that you can legally obtain ought to be a right for anyone. After all, i am allowed to learn off anything i can legally read (and perhaps even illegally read). The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.

> being allowed to train on any data that you can legally obtain ought to be a right for anyone.

I have the opposit viewpoint to the extreme. They shouldn't be allowed to even read that data until they are very clear about what they will or not do with it.

Can they publish it? Can they store it? Can they use the information in it on prediction markets? Etc.

Humans reading texts historically come with little negative consequences, but machines reading and processing texts en masse is more dangerous and should be regulated.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#306
>in areas where there is an arbitrary choice between particular words or terms within the code, the watermark can be used, such as comments within code.

I wonder if this is why Opus 5 keeps writing excessively long comments, even though I keep instructing it not to (both in chat, CLAUDE.md, and in its memories)

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#308
post #51

>I want any LLM I use to choose the very best, most precise words at every single decision point. Does the author think he is currently getting T=0 output from Claude? Is he under the impression that T=0 produces the "best" writing? This entire article just seems so detached from the basics of how LLMs work.

I think the author is just mad people will be able to detect and filter out their AI slop writing in the future.

Gruber has long been a talented, excellent writer. I hugely doubt he uses AI at all, nor does he plan to.

His first take on this situation was cutely naive, thinking they were going to inject secret hidden unicode characters. But ultimately he has a massive hate on for the EU -- they were mean to Apple once -- and it comes out in any topic that overlaps.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#309

Earlier quoted context omitted.

It's a bit different when "training on any data" means basically storing a lossily-compressed copy of that data, that could be spit out years later if the model decides to do so.

It's exactly the same problem as with humans, though. It's part of why we sign NDAs, and why their duration is measured in years (and that's not even targeting the human retention - just duration after which information ages enough that its disclosure is not likely to negatively impact anyone who cares).

It's not exactly the same problem, in that you can parallelize usage of an LLM and copy it over to another computer, but cannot do the same things with a brain. Put it another way, humans do not have the processing power needed to answer hundreds of millions of queries per day, while LLMs do.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#310
post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

Why not switch it around? Solves problems of privacy (no data upload), is far more doable and it's far more important to be able to verify content has not been tampered with after originating from a human or capture device (like a camera). We also have decades of cryptographic experience in reliably signing text, images, video, etc. and it doesn't break apart because a text is too short.

Having proof that content (especially images and video evidence) is unmodified (whether via Photoshop, Paint or a model) is far more valuable then having evidence that an image was manipulated or generated fully by a model (which still leaves other forms of manipulation), I feel the same goes for human authored vs generated text. Free to admit that using models to generate any kind of media whole-cloth is still unappealing to me and I still pay for commissioned artwork or make it with my limited abilities for what that's worth. Do like to (poorly) write my musings too and see UX as something were thoughtful contributors (like the opinionated, sometimes controversial, but certainly talented GNOME Gitlab contributors) can make a major impact.

Code can be beautiful, interesting and serve purpose beyond execution, of course, but for most people, in most cases, it does not in the same way as audiovisual content (not limited to art). Having code just to execute and resolve a problem can have value all in itself, the code being a means to an end whose quality, let us be honest, was barely a concern in most corporations long before LLMs.

Also have rarely (honestly never) before LLMs fully owned all parts of any code base, always relied in part on someone's prior effort in (Flutter/Dart mostly) packages, whereas when writing, drawing, etc. I have far more situations where I make something from scratch and everything there is only there because of my conscious decision. Even simple marketing mockups that, quality wise, any modern model would beat feel different when I was fully in control, where to place what, etc. Objectively worse (at my skill level), probably, but still never the same.

Knowing something was made from scratch by a human has value to me, beyond misinformation prevention. Knowing for a fact that LLMs were used instead of importing a library, using a template, or something similar that leads to expending similar amounts of effort, I don't see that being nearly as valuable. Heck, with all the importing and my experience back then vs now, I am spending more effort actually fully reading any LLM output in my code then I spent back then auditing Flutter/Dart packages. Then again, LLM output fails far more unpredictable then those messy packages that simply got Gradle to take down my system...

Happy to admit, I have been skeptical of watermarking LLM output being feasible for quite some time and having looked into SynthID Text and proposals being researched, I am convinced that it is challenging to impossible beyond the lowest common denominator and less important then proofing human authorship.

It will catch people just copying LLM output into their replies without thought, which is not a negative in my book, especially if it is not discernibly affecting output quality in regular use cases. Anyone who wouldn't copy Wikipedia into their dissertation will, in my opinion, be able to bypass text watermarking as proposed however, I feel we need to be honest there.

Thing is, if that's the case and text watermarking will only ever catch LLM created slop, is that a bigger problem then the misinformation, harm to creators due to authorship questions and accusations, making it harder to use evidence in proceedings, teachers not trusting students even when they did the work themselves, etc.? Signatures for all such cases will be difficult to implement, yes, but I feel are going to be of greater value in the not to distant future and I equally feel are not impossible, not least because idiots will always want to hide their LLM usage, whereas human authorship is something they take pride in and want to proof.

Post reply on HN