So Anthropic goes on the shit list with Sony Blu-ray and its cinavia watermarking.
Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
621–630 of 776 posts
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#622Earlier quoted context omitted.
I wonder if that is entirely true. It is also in the AI companies' own best interest to be able to detect AI generated text so that they can avoid training on it ("Habsburg AI"). In the case of Anthropic it would also be entirely unsurprising if they've been lobbying the government to force everyone to do something in their (Anthropic's) own best interest.
> It is also in the AI companies' own best interest to be able to detect AI generated text so that they can avoid training on it It’s also in their interest to demonstrate that they can be trusted and to show that they at least pay lip service to limit the obvious downsides of the tools they are selling. The use cases they sell to mainstream audiences are not affected by detection tools. The point of having a LLM do…
The EU mandate for AI watermarking is likely the first step in the direction of prohibiting AI for specific use cases. The pretext for outlawing (or at the very least controlling) the use of AI when the time comes will be something along the lines of data integrity or just general compliance legalese.
So, the use cases that are being sold to mainstream audiences actually will be affected by detection tools, especially if the output is intended to be monetized in some way. In the near future the EU will likely come down with heavy intervention to prevent AI from impacting employment rates across Europe. The number of legitimate use cases for costly frontier models drops significantly once eliminating professional jobs is off the table. This is all conjecture at this point though.
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#623Earlier quoted context omitted.
That's inaccurate in two ways: (1) The behavior that is approximately what you describe is not "fundamental" (though it may not be something you can disable on some hosted providers), it is an option that is not fundamental (and with runtimes where you have full control can be either disabled or tuned in a large number of manners), and (2) The actual behavior that is approximately what you describe already usually in…
You're generating a pseudo random number one way instead of another way. How would that inherently compromise quality?
These are services, so "how would do this today?" is irrelevant.
The real question is: "What can manipulation of pseudo-random number generation do?"
We know that in the cryptographic world, attacking "randomness" is a key offensive capability. It will be here as well -- if Anthropic can watermark text as generated it's LLM, will it be able to watermark outputs as generated by "Spooky23/FooCorp"? Can I pay Anthropic to steer inquiries in a way that benefits my company or governemnt?
Pseudo-random to the end user appears random. Most treat it like a random chance. It is not.
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#624I’m surprised by the comments here being so favorable to anthropic. The comments are right about there being no “best” token, and yeah Gruber may have an agenda here. But I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. Take the “gray” and “overcast” choices. And lets say before applying synthid the percentages were 48% and 52%. Those pe…
> I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. But it doesn't! The distribution doesn't change at all. The only thing that changes is that sampling of that distribution becomes deterministic as per a precomputed seed.
You could describe that as taking an input distribution and a sampling procedure and producing an output distribution. This is a difference in sampling procedure that produces a deviation in the output distribution.
(If you don't like calling it a distribution when it's at 100% for the chosen token and 0% for all others, then look at it as an output distribution across all possible prompt inputs, or perhaps just the cluster of prompts that achieve whatever you're trying to accomplish.)
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#625Earlier quoted context omitted.
> especially not for idiotic reasons like facillitating AI stigmatization. From the people I’ve talked to at universities, LLM based cheating in education is an unstoppable nightmare. I don’t have a problem with LLMs. But I do want the cheating to - somehow - stop. The people who cheat miss out on learning. And the people who don’t cheat have their degrees devalued by those who do cheat.
> The people who cheat miss out on learning. They aren't there to learn. They are there to jump through hoops to get a degree that will let them get a job so they can make money and prosper . The learning is entirely secondary. The cheating will stop when there is no longer any economic incentive to be there in the first place. People with "pure" motives will refuse to cheat on their own, precisely because they want…
Easy to say, hard to come up with a believable alternative. In the meantime, you're flunking out a lot of people for having the integrity to not cheat and as a result not being able to keep up with an artificially inflated workload.
You can't just destroy some signal and handwave that you'll make it up in some other way.
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#626Crazy how a smart person like this fails to understand the gumbel softmax technique. It does not affect writing quality at all, provably. The very fact that there is generally no "best next token" with 100% certainty is precisely why the trick works (you cannot watermark a response to "respond with the To be or not to be soliloquy from the first folio Hamlet", for precisely this reason).
Gruber is always happy to lead with his emotions and backfill justifications for them. See his recent debacle with App Store review: https://daringfireball.net/2026/08/retraction_app_store_reje...
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#627Crazy how a smart person like this fails to understand the gumbel softmax technique. It does not affect writing quality at all, provably. The very fact that there is generally no "best next token" with 100% certainty is precisely why the trick works (you cannot watermark a response to "respond with the To be or not to be soliloquy from the first folio Hamlet", for precisely this reason).
> a smart person like this
Can you give an example of something smart John Gruber has said or written? Because I can't think of one, but I can think of many dumb ones.
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#628Earlier quoted context omitted.
> To change those percentages to 45% and 55% This seems like a fundamental misunderstanding of how this sort of watermarking works. (Either that, or I have a fundamental misunderstanding of how it works lol.) It doesn't change the probability distribution of the next token at all. If you were getting XYZ 48% of the time before, you're still getting XYZ 48% of the time. What's changed is where the random numbers come…
huh, thanks for commenting this! I trusted the linked explanation post https://declaude.org/watermarking/ but actually reading the the synthid paper showed me that my understanding was wrong: https://www.nature.com/articles/s41586-024-08025-4 It is definitely blurrier whether you can say this approach changes the distribution then. By definition, it _has_ to change the probabilities of output tokens, but it's not tot…
You’re not getting it. The probabilities do not change at all. The only change is given some probabilities there is a deterministic method for determining which symbol was sampled from that distribution. The distribution or sampling process itself is not modified.
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#629Earlier quoted context omitted.
Yep, I literally ask it to say it in 20 different ways. Sometimes a sentence structure or a combination of words will just work better. The goal is usually to simplify a sentence without losing meaning. I don't expect the LLM to read my mind. The unit of work is too small for intent to matter, and I'll just steer the next recommendations in a direction as needed. Most of the suggestions are crap, but they can contain…
And this beats just writing the piece yourself?
How did you understand this as not writing the piece myself?
Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing
#630Gruber shows here that he really doesn’t understand the basics of how LLM text generation works. It’s weird he picked this battle about the quality of writing in LLMs. Was he planning to use LLMs to write his articles? Well, not that weird actually. He just has a hard-on against anything that comes from the EU since Apple got in trouble. If the EU said tomorrow that they want peace in the world he’d be in Fox News th…
Gruber went from the naively wrong claim that it would insert secret hidden characters (which would be trivial to remove, obviously), to quickly writing a giant essay as if he's an expert on LLMs. Like you said, he is strangely fixated on the EU, and is certain any EU rule is the worst thing in the universe, and this whole piece seems motivated by that guiding force. Further he later compares Gemini to Anthropic mode…
How is he "strangely" fixated on EU? The EU tries to limit Apple's power, Gruber shills for Apple, Gruber is against the EU. Nothing strange about it.