Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

151–160 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#151

Post by beering »

dofm wrote: post #77
Gigachad wrote: post #66

Earlier quoted context omitted.

I think the author is just mad people will be able to detect and filter out their AI slop writing in the future.

This is not it, no. He is not using AI and it is not I think remotely in his nature to surrender that control. He is engaging with this on principle. Again I am not sure I agree with him, but then it’s a hypothetical because I am not going to get an LLM to write for me either.

Well, it cant be that he is super worried on behalf of people who publish AI slop. That’s not a credible motivation. In fact, he complained a lot about the new ChatGPT app so I can’t believe your claim that he is not using AI.

Seems like he really likes to use LLMs and is worried that quality will be degraded. But he will never demonstrate such degradation scientifically, we don’t have anecdotes even.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#152

Post by robomc »

wpietri wrote: post #78
levocardia wrote: post #26

Crazy how a smart person like this fails to understand the gumbel softmax technique. It does not affect writing quality at all, provably. The very fact that there is generally no "best next token" with 100% certainty is precisely why the trick works (you cannot watermark a response to "respond with the To be or not to be soliloquy from the first folio Hamlet", for precisely this reason).

It seems to me like he started out mad and looked to justify it. I'm skeptical that anybody generating LLM text is really all that concerned about optimal word choice. Or even particularly good prose. But let's pretend that person exists. If that person tried, say, an open model and that same model with watermarking applied, I'd be eager to hear their thoughts on the prose quality. Especially if they built an experim…

I assume they are just passing off AI prose as their own and don't want anyone to be able to tell. Which is surprising for someone who's been blogging for a thousand years. But I don't really see any other reason for this amount of heat and FUD.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#153

Post by dofm »

beering wrote: post #145
dofm wrote: post #82

Earlier quoted context omitted.

It’s fully possible I didn’t explain it very well in the first place, but he is making a wider point. The point I made (quite briefly) is that watermarking is only feasible because for good writing it is necessary to use T>0, or the writing will never explore a more creative choice, and that at T=0 you don’t even need a watermark to spot LLM-generated text. The point he is making is consistent with this, isn’t it? Ei…

No, you are jumping to conclusions about how watermarking works. This is some audiophile thinking that because your RNG is “pure”, you get text with an expansive soundstage or whatever. Intuitively this may be true or false depending on your personal prior but you’d need to show it mathematically. The overall token distribution shouldn’t change and the frequency at which you see the word “load-bearing” will remain th…

> This is some audiophile thinking that because your RNG is “pure”, you get text with an expansive soundstage or whatever.

You are projecting that onto me, and I cannot tell you how comically poorly aimed it is.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#154

Post by akersten »

Respectfully, you are all missing the point.

Watermarking is bad not just because of the principled stance that your tool should not be working against your own interests (the passionate argument in TFA), but specifically because it lends credence to the idea that AI detection is a valid and possible thing to do perfectly.

As technologists of course we know "oh well yes but with some confidence interval we can detect AI token bias across a large corpus of text." To JimBob in charge of publishing your paper or reviewing your PhD submission, all he knows is "anthropic says AI detection is possible so this 30% chance your paper was written by AI means you've plagiarized." Do you really think you're winning the argument with the certified, law-approved plagiarism detection machine? No, you're not, and your career is over.

It's irresponsible to develop watermarking because it is not anywhere close to a perfect science, but it will be treated like one by people with the power to ruin your lives. Even if you've never touched AI in your life, your paper is going through the "maybe it says you cheated" box, and you better hope those dice don't come up snake eyes.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#155

Post by breezybottom »

aselimov3 wrote: post #14

This article feels slightly incoherent. You want high quality precise writing and to use an LLM to generate it? Feels like those are diametrically opposed

This feels like the inevitable outcome of a STEM-only education system. Now people think there's a mathematical formula for picking the "best" words, instead of having to be thoughtful and creative.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#156

Post by FeteCommuniste »

inigyou wrote: post #97

> One of my fundamental problem with this is that no two synonyms carry the exact same meaning. “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter. Then why are you using an LLM to write? They're not capable of understanding such nuance. They do pick randomly between two…

Yeah, I snorted at the sentence "The exact words we choose when writing matter." Well, then why the heck are you using an LLM to "write," man?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#157

Post by richardatlarge »

epihelix wrote: post #50
smallerize wrote: post #7

Translation: No one can ever again use Claude for proofreading their own prose unless they’re willing to risk that the whole thing might be flagged as having been generated by Claude. I think that was intended, yes.

You know, back in the era when proofreaders were human, I never one met a proofreader who rewrote my text afresh, rather than annotating the text with a pen. It's still possible to use Claude to proofread - highlight grammatical, flow, structure, logic errors and make simple suggestions for you to pick and choose or adapt as you wish. No watermarking will flag your text. No flaw accusations of LLM authorship will hau…

Nonsense. Forget proofreaders. Think editors. In publishing some editors practically wrote the books. And then theres ghostwriting ! Think of that!

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#158

Post by danofsteel32 »

npilk wrote: post #19
syrrim wrote: post #9

> I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo ran…

I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want. On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense. Even good human writers would probably struggle with this constraint. It wou…

I tried brainstorming what an agent harness for writers would look like.

https://chainofbranches.com/conversations/2/branches/20/

I’m not convinced it’s possible. A good nights sleep and a notepad in a quiet room still feels like the state of the art toolchain for writers.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#160

Post by wasabi991011 »

Art9681 wrote: post #73
levocardia wrote: post #26

Crazy how a smart person like this fails to understand the gumbel softmax technique. It does not affect writing quality at all, provably. The very fact that there is generally no "best next token" with 100% certainty is precisely why the trick works (you cannot watermark a response to "respond with the To be or not to be soliloquy from the first folio Hamlet", for precisely this reason).

If this is true then the probability of the detection tools flagging completely human generated text as AI generated is non-trivial. Let's say I write a completely original piece and the detection tool says there is a 36% probability it was generated with Claude. What then? Now it's up to the person looking at the score to cast a subjective judgement. Maybe to me, anything over 25% is unacceptable. Maybe to someone e…

That's an interesting problem to discuss, but unfortunately TFA spends no time discussing that.
Post reply on HN