Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

631–640 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#631

Post by a2ff6eeb0 »

mrngld wrote: post #404
Topfi wrote: post #377

Earlier quoted context omitted.

Fair point, for text it is far harder to prevent signatures being applied to generated text vs images at the moment of capture and most approaches I can come up with to remedy this can either be bypassed (edit histories can be output by models similar to humans) or will be controversial. Taking a page out of the anti-cheat textbook, mainly written for gaming, there are methods which might hold in the medium term. Les…

I've been thinking for a while that all of this is just trying to grasp tighter the last bits of sand escaping between our fingers. The end game, perhaps, is trust. Do you trust or know the source? If you don't, assume it was AI generated. If you do, accept it as authentic based on whatever they disclose, but know that it's possible they aren't being totally honest or were themselves fooled in some way, depending on…

Well, the end game is that AI is better at thinking us, and having a human brain involved is a net negative. The goal is to let the AI do the toil if thinking for us, and we can get the rewards.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#632

Post by Nimitz14 »

simonh wrote: post #237
tempestn wrote: post #230

LLM output, as the author acknowledges here, is already non-deterministic. Next token probabilities are set, and tokens are chosen pseudo-randomly. As I understand it, this watermark is just going to be a matter of using a known seed and algorithm to make those pseudo-random choices, such that a signature can be detected. The important thing is, it's not replacing intentional choices with random ones, it's just gener…

That cannot be true. The quality of an LLM's output is the quality of the probability calculations for the next token. Anything that degrades the relationship between the system's best assessment of the appropriate probability and the actual probability used is a degradation of the quality of that probability and therefore of the output. If this didn't have a detectable effect on the quality of the token probability…

You have some fundamental misunderstandings on how LLMs work.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#633

Post by 0-_-0 »

recursive-call wrote: post #367
voidUpdate wrote: post #199

> "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much

The exact words matter to people who people who don’t use it to write for them. A lot of people use Claude as a friend/therapist/romantic partner/etc. That’s who I imagine would be most affected

Affected by what exactly?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#634

Post by sfink »

Imnimo wrote: post #51

>I want any LLM I use to choose the very best, most precise words at every single decision point. Does the author think he is currently getting T=0 output from Claude? Is he under the impression that T=0 produces the "best" writing? This entire article just seems so detached from the basics of how LLMs work.

This is a reductionist counterargument. Sure, the passage you quoted does sound like he's being equally reductionist. But the underlying point does not depend on T=0. You could state it as saying that instead of minimizing error (maximizing "writing quality"), you're using some of that error for watermarking and minimizing the rest.

Describing it in terms of a word-by-word choice is simpler, but writing quality is dependent on the interplay between words.

"The weather today was cold and {grey,overcast}." If the next sentence is "I miss yesterday, when it was {bright,sunny}." then the choice between "grey" and "overcast" is no longer neutral. "grey" and "bright" pair together, as do "overcast" and "sunny". Or if you disagree with my aesthetic sensibilities, consider:

    The weather today was cold and {grey,gray}. The {color,colour} of the sky matched my {humorless,humourless} mood.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#635

Post by khelavastr »

It's because the individuals who write those laws are literally trying to shove neo-Nazi policies into EU practice. I wonder how many people would make policies like this if they and their families were publicly identified and criticized as neo-Nazi elements in society.

You think someone will write Nazi-promoting AI policy like this when society is encouraged to look at their families as examples of neo-Nazi corruption? When their wives' and kids' friends spurn them while their families engage in obvious criminal activity to harm world productivity?

Critics need to be more precise.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#636

Post by teeray »

kevincox wrote: post #353
zrm wrote: post #293

Earlier quoted context omitted.

> Any stenographic system that you have the code for can be trivially defeated. They're giving you an oracle regardless, which is almost as good. Take LLM output, make some modification, ask the detector if it's LLM output, repeat until you learn what kind of changes you have to make to defeat it. Or don't even bother learning what to do, just make arbitrary changes until it says it's not, so when the person they're…

I assume this oracle will be behind 20 layers of anti-bot protection, CAPTCHAs and hardware attestation challenged. It will be incredibly painful to use. It won't stop the motivated attackers, but will make it too annoying for the average person.

When all else fails, you can hire a lot of folks cheaply to effectively Mechanical Turk it with their home internet connections.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#637

Post by kergonath »

sailfast wrote: post #596
kergonath wrote: post #522

Earlier quoted context omitted.

Very intelligent people should be able to grasp that a LLM is not intelligent.

Have you tried to "rubber duck" an idea with an LLM. The latest models are pretty damnably good at it.

I have, and I still do this occasionally. I don’t believe the plant on my desk (I don’t have a rubber duck) is intelligent. I don’t need it to be intelligent, either.

I also do it with a LLM every now and then and, while it’s feedback is more useful than a toy’s, it does not need to be intelligent either.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#639

Post by sfink »

Spooky23 wrote: post #618
sfink wrote: post #609

I agree with the substance of this article, and disagree with the author's reaction to it. The part I agree with: It is true that watermarking can be done by "just" swapping one PRNG for another, and it is even true that with today's LLMs, it is possible that this will not degrade the output. But it has a cost, and as things improve, that cost will matter. You are intentionally reducing the degrees of freedom in the…

> So I agree that the watermarking has a cost. But you can't leave out that it is an attempt to reduce negative externalities of AI. Whether it's a realistic or worthwhile attempt is a whole other debate (and Gruber does a good job of debating just that in the latter part of the essay), but saying that the user's needs are the only thing that should ever be considered is reprehensible. Computers are tools that exist…

I agree with you as long as things are at a small scale. But at a large scale, these things reshape society and what it means to be human. Social networks started out[1] as almost wholly good. The negative effects came from scale. Same with advertisement-funded websites and tons of other things that started out as being overall positive for the commons and ended up being highly negative. You can't just stick your fingers in your ears and ignore what is happening in reality.

> Anthropic is crowing about this achievement because they are afraid of the dirt cheap AI models coming out of China and eventually other places impacting their valuation. Full stop.

It's the "full stop" that I'm disagreeing with. Yes, you can make valid arguments about the motivations behind this, or the effectiveness of it, or whatever. You can even conclude that it's a net negative. That's my current leaning. But "computers are tools that exist to serve" is an excuse, a conscious decision to abandon responsibility. Hammers, nails, social media, biological weapons, and date rape drugs are tools that exist to serve.

[1] Ok, fine, at least one started out as a sleazy way to talk about people's physical attractiveness and behavior anonymously with repercussions, in a way where the victims would have no meaningful recourse. For the sake of my argument, pretend we're talking about Friendster instead of Facebook, please?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#640

Post by matheusmoreira »

sfink wrote: post #625
matheusmoreira wrote: post #482

Earlier quoted context omitted.

> The people who cheat miss out on learning. They aren't there to learn. They are there to jump through hoops to get a degree that will let them get a job so they can make money and prosper . The learning is entirely secondary. The cheating will stop when there is no longer any economic incentive to be there in the first place. People with "pure" motives will refuse to cheat on their own, precisely because they want…

Easy to say, hard to come up with a believable alternative. In the meantime, you're flunking out a lot of people for having the integrity to not cheat and as a result not being able to keep up with an artificially inflated workload. You can't just destroy some signal and handwave that you'll make it up in some other way.

> the integrity

There is none. It's just a jobs program fueled by student loans. Higher education in the west has been corrupt for quite a while now. AI is just the final nail in its coffin.

Post reply on HN