Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

431–440 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#431

Post by applicative »

ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

I’m worried that until such tech is perfected, pervasive and uniform across all models, education will be dead, as it certainly is at the moment. Flat out dead.

I read stacks of term papers all year and it is a reality that, apart from such schemes, we are in an extinction event for civilization.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#432

Post by PunchyHamster »

ghrl wrote: post #246

My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illici…

I expect them to work as reliably as AI text generators do now...

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#433

Post by matheusmoreira »

mangoman wrote: post #428

I’m surprised by the comments here being so favorable to anthropic. The comments are right about there being no “best” token, and yeah Gruber may have an agenda here. But I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. Take the “gray” and “overcast” choices. And lets say before applying synthid the percentages were 48% and 52%. Those pe…

> But I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model.

This. I want the model I'm paying for to be "pure". I don't Anthropic or anyone else messing around with it, especially not for idiotic reasons like facillitating AI stigmatization. The "safety" nonsense is obnoxious enough.

They should train the best possible model and let the weights speak for themselves, not degrade it into some perverted form to appease people who hate AI anyway.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#434

Post by andy_ppp »

As the algorithm tries to keep the stenography in place this will mean if you specifically ask for a different phrasing of one paragraph, other parts of the document will need to change to keep the supposedly impossible to detect AI watermark in place? Won't people also just quickly do analysis on this to figure it out and remove the watermark - it's hard to do this for individual messages but when you can create an infinite number of messaging to train on, I'd think analysis of how the signal works will be quite trivial.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#435

Post by applicative »

bsenftner wrote: post #374
unbelievr wrote: post #361

Earlier quoted context omitted.

The paper for it is open. The technique isn't really hiding information in the text itself, but by forcing some of the rolls to follow a specific pattern. LLMs work by estimating the most likely next token, so there's sometimes a list of possible candidates that would all work in the text (e.g. synonyms). At low "temperature", the output is a bit more deterministic and otherwise it's a weighted dice roll of which tok…

In practice, it is theater. Are they going to do this with the code output too? This is nonsense security theater for the low thinkers to have a sense that someone is in charge. When we all know nobody is in charge, anywhere.

EU regulations are going to force it, friend. The tsunami is coming and cannot be stopped and Anthropic has jack to do with it.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#436

Post by cbondurant »

> I want any LLM I use to choose the very best, most precise words at every single decision point.

Oh! If you want that, you should run your own model and set the generation temperature to 0 :) Because that's not what any commercial LLM is doing. Never has been. This is just making up a universe that doesn't exist so you can get mad about no longer being in the universe that doesn't exist.

The masking technique of using a subset of the statistical distribution for each next token isn't going to be meaningfully distinguishable from a natural language perspective. I honestly think its a very elegant way to implement watermarking. I've got no real opinions on how effective it will be to people actively trying to defeat it, but I suspect that the people who are trying to pretend that LLM text was something they wrote themselves are probably too lazy to put in the work to try and defeat it anyway.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#437

Post by Laurel1234 »

mrweasel wrote: post #231

This seems like a non-issue, or maybe I have the wrong expectations about writing. You write a text, ask Claude to proof-read it, but then you wholesale just copy Claudes output and use that as the final text? Wouldn't you review the changes it suggests and only take those you agree with, there by completely bypassing the watermarking? Alternatively, you ask Claude to write the whole thing and proof read it yourself.…

[dead]

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#438

Post by _kulang »

einpoklum wrote: post #402
_kulang wrote: post #32

Earlier quoted context omitted.

I moved to Sol for my writing and it is so so much better. But it makes more mistakes. I think they have different ideas of product but it seems OpenAI is going to follow Anthropic’s lead over the next year. I think I am going to put more effort into my writing skills to remove myself from this awful situation

> I moved to Sol for my writing and it is so so much better. No it's not. The bad part about it is that some machine is writing instead of you, not the specific stylistic idiosyncracies.

Sure – but I’m entitled to make a judgement on what I consider to be “good” and “bad” output from an LLM, where “good” just means “helpful to my process”.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#439

Post by zoobab »

0x_rs wrote: post #101

I'd encourage reading this paper, and literature on scaling laws in autoregressive models: https://arxiv.org/abs/2303.11156 Total variation distance has been measured to decrease as you scale a model, and that is the primary mechanism "watermarking" as discussed in the Anthropic announcement relies on. It becomes more difficult to reliably detect text as a fixed sample count without tweaking the distribution further.…

"logging solutions"

In 2005, FFII predicted that the Data Retention directive would not pass the Courts.

It just took 10 years for the CJEU to strike down the measure, as it is "mass surveillance".

Logging everything your bot does by law is of the same dimension?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#440

Post by josephg »

mangoman wrote: post #428

I’m surprised by the comments here being so favorable to anthropic. The comments are right about there being no “best” token, and yeah Gruber may have an agenda here. But I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. Take the “gray” and “overcast” choices. And lets say before applying synthid the percentages were 48% and 52%. Those pe…

I’m sure it matters. But how much does it matter? How much (perceived) intelligence would you be willing to sacrifice for an accurate AI predictor?

I’d sacrifice a few %, easily. Maybe 10%. The models are getting smarter at such a fast rate that I’d be willing to lose a month or two of progress to help slow down the AI cheating epidemic.

It sounds like you expect this fingerprinting approach would dramatically reduce the intelligence of their models. But I’m sure anthropic has measured it. I doubt they would have rolled this out if the intelligence cost were that significant. I suspect the cost is less than 5%. I personally can’t tell any difference from before they added fingerprinting. I bet you can’t either.

Post reply on HN