Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

551–560 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#551

Post by tmp10423288442 »

recursive-call wrote: post #367
voidUpdate wrote: post #199

> "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much

The exact words matter to people who people who don’t use it to write for them. A lot of people use Claude as a friend/therapist/romantic partner/etc. That’s who I imagine would be most affected

LLMs never are limited to "exact words", because their output is inherently probabilistic. The method Anthropic (along with Gemini, who has been using the exact same watermark for at least a year) uses doesn't bias the output token distribution, just reseeds the PRNG in a way that can be detected after the fact: https://www.anthropic.com/news/claude-text-watermark#which-s...

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#552

Post by tmp10423288442 »

red_admiral wrote: post #192
syrrim wrote: post #9

> I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo ran…

The best discussion I've seen so far is from Scott Aaronson: https://scottaaronson.blog/?p=6823 > To illustrate, in the special case that GPT had a bunch of possible tokens that it judged equally probable, you could simply choose whichever token maximized g [a cryptographic function]. The choice would look uniformly random to someone who didn’t know the key, but someone who did know the key could later sum g over all…

Scott Aaronson basically invented this form of watermarking, so it's not surprising he has a good description of it.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#553

Post by logicchains »

applicative wrote: post #431
ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

I’m worried that until such tech is perfected, pervasive and uniform across all models, education will be dead, as it certainly is at the moment. Flat out dead. I read stacks of term papers all year and it is a reality that, apart from such schemes, we are in an extinction event for civilization.

Education is in the best place ever for people who actually want to learn, they can have 24/7 access to a tutor with a wide breadth of knowledge and infinite patience for stupid questions. Education is in a terrible place for people who just want to get a degree and don't care about actual learning, but such people generally don't contribute much anyway, and are the easiest to replace with AI, so no big loss.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#554

Post by atroon »

ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

>Any university using AI detection in their submission pipeline, or lawyers, editorialists, proofreaders that check for AI marks will be sending significant amounts of text like unpublished research, books, potentially internal documents and more, most of which is high quality human written, to dozens of AI companies, blindly trusting they won't train on any of that.

Uh, yeah, why do you think it's setup this way? The frontier companies desperately desire more high quality human text and this is how they are planning to get it for free.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#555

Post by Tepix »

KaiserPro wrote: post #470
Sharlin wrote: post #408

Earlier quoted context omitted.

MP3 is lossy. FLAC is lossless. So obviously a certain type of people are going to make a religious war out of it.

Essentially yes. Flac takes a raw .wav and effectively zips it up to shave off a certain amount of space. (there are nuances, I think the compression scheme is designed for streaming.) mp3 is perceptual, so throws away the stuff that humans can't hear. This yields a much smaller file. However its all a sliding scale like PNG vs jpeg. a .jpg with a quality setting of 85 will be almost identical to a .png in visual qua…

Interestingly, in a test c't magazine did a long time ago, the test audience preferred mp3 256kbit over the original.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#556

Post by jjk166 »

pjc50 wrote: post #544
jjk166 wrote: post #538

Earlier quoted context omitted.

Seems like exactly the sort of problem the threat of defamation lawsuits are meant to solve.

Evidence is a massive problem here. As well as the extremely high threshold for US defamation; political candidates routinely tell the most absurd lies about each other.

The high threshold is only for public figures, which must show actual malice. For everyone else (e.g. your typical student falsely accused of plagiarism), they only need to show the statement was false, published, targeted the plaintiff, and caused damage - it's still defamation even if they were simply negligent in checking the facts. The company might be able to cover themselves if they are very clear they are just giving a statistical probability and not saying their model was definitely used, but if they imply they have hidden information that points to their conclusion (which is kind of the whole point of their algorithm) then that's not a valid defense. I can't say they would lose in practice, but in theory they ought to lose.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#557

Post by katatue »

applicative wrote: post #452
vintermann wrote: post #379

Earlier quoted context omitted.

A way to think of it is that any time the model faces a choice, it leaks some fractional bits of information. Instead of making the choice randomly, you can put information in those bits. But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty. Exotic prompts ("Use no more than five E's in four consecutive words anywhe…

In fact such ‘arbitrary’ constraints uniformly improve composition. Thus eg if I force a - largely arbitrary - technical glossary to be unrelentingly applied to a translation, every single sentence improves in quality.

Oulipo vindicated.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#558

Post by solid_fuel »

redfloatplane wrote: post #286
carlosrg wrote: post #270

Gruber shows here that he really doesn’t understand the basics of how LLM text generation works. It’s weird he picked this battle about the quality of writing in LLMs. Was he planning to use LLMs to write his articles? Well, not that weird actually. He just has a hard-on against anything that comes from the EU since Apple got in trouble. If the EU said tomorrow that they want peace in the world he’d be in Fox News th…

Yes, I decided to stop reading his blog relatively recently after some extremely hot takes on EU policy. I don't feel his thoughts on the matter are particularly well-thought-out, and I feel like he's just stanning for Apple from his priors rather than from any grounding in reality. I dunno, I guess that's what you should expect from Gruber but these EU-bashing articles lowered the enjoyment I got from his blog under…

Freaking out about basic EU privacy and safety measures is always one of the biggest red flags that someone is becoming a politically motivated hack, especially when that someone lives in the US where there is an active fascist movement tearing apart the government.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#559

Post by declan_roberts »

I really don't think Anthropic has the leverage to pull this kind of thing off in the face of competition. OpenAI doesn't do it (yet), Grok doesn't do it, neither does Deepseek. I'm sure the latter two have plans to intentionally never do it.

Who is even asking for this? Sounds like something some obsessive internal employees would push on the world.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#560

Post by yorwba »

neuroticnews25 wrote: post #275
levocardia wrote: post #26

Crazy how a smart person like this fails to understand the gumbel softmax technique. It does not affect writing quality at all, provably. The very fact that there is generally no "best next token" with 100% certainty is precisely why the trick works (you cannot watermark a response to "respond with the To be or not to be soliloquy from the first folio Hamlet", for precisely this reason).

>It does not affect writing quality at all, provably Prove it, then? It's not a claim that GumbelSoft paper makes: "Regarding generation quality (perplexity), GumbelSoft shows relatively low perplexity" https://arxiv.org/html/2402.12948v3

They do claim that the per-token output distribution remains unchanged, but the proof is relegated to Appendix B.1. The perplexity comparison includes methods that do change the output distribution.
Post reply on HN