Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

191–200 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#191

Post by lazide »

npilk wrote: post #19
syrrim wrote: post #9

> I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo ran…

I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want. On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense. Even good human writers would probably struggle with this constraint. It wou…

Human writers do better because they can think, and adjust, based on context.

They are also usually worse (which is often better!) because they are usually lazy and don’t want to spend effort they do not have too, to accomplish their goals.

Their goals are often complex and nuanced.

None of this is true of LLMs.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#192

Post by red_admiral »

syrrim wrote: post #9

> I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo ran…

The best discussion I've seen so far is from Scott Aaronson: https://scottaaronson.blog/?p=6823

> To illustrate, in the special case that GPT had a bunch of possible tokens that it judged equally probable, you could simply choose whichever token maximized g [a cryptographic function]. The choice would look uniformly random to someone who didn’t know the key, but someone who did know the key could later sum g over all n-grams and see that it was anomalously large. The general case, where the token probabilities can all be different, is a little more technical, but the basic idea is similar.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#193

Post by lesostep »

npilk wrote: post #19
syrrim wrote: post #9

> I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo ran…

I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want. On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense. Even good human writers would probably struggle with this constraint. It wou…

I also really liked the quote "Your existence is not impossible, but it's also not very likely" from the Night Vale podcast.

I feel like the existence of good writing is also not impossible but not very likely, and so of course LLM can only write mediocrity, even when taught only on great writing.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#194

Post by dragonwriter »

reliablereason wrote: post #28
dragonwriter wrote: post #15

Earlier quoted context omitted.

That's inaccurate in two ways: (1) The behavior that is approximately what you describe is not "fundamental" (though it may not be something you can disable on some hosted providers), it is an option that is not fundamental (and with runtimes where you have full control can be either disabled or tuned in a large number of manners), and (2) The actual behavior that is approximately what you describe already usually in…

(1) LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time. This is a consequence of how they are trained.

> LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time.

LLMs are likely to get stuck even with sampling if asked to generate tokens on their own long enough, though sampling does tend to stretch out the time before that happens (as do other techniques that don't involve sampling, like applying repetition penalties directly to token logits). But LLMs generally aren't left to infinitely extend their own output, and the length response typically needed in the use case is much shorter than the would result in collapse given the kinds of inputs expected in that use case, the existence of the theoretical eventuality may not really matter.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#195

Post by ACCount37 »

robocat wrote: post #161
Maxatar wrote: post #119

Earlier quoted context omitted.

>There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”. You're mixing up two claims here, and only one of these is kind of true. Yes LLMs do internally plan ahead in a way that is emergent rather than strictly part of their architecture, so that part of your claim is true. The way you word it by saying they are "coalescing the probabilities of a range of t…

What a crazy link: So we train a second copy of Claude to work backwards—reconstruct the original activation from the text explanation. We consider an explanation to be good if it leads to an accurate reconstruction. We then train Claude to produce better explanations according to this definition using standard AI training techniques. Incentives to train a pathological liar. There's no baseline so can only catch out…

[flagged]

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#197

Post by ilogik »

slhck wrote: post #190

The fact that Gruber points to a "must-read" article about how the watermarking works, which, in turn, was very obviously entirely LLM-generated, says a lot about his lack of experience reading LLM output. If Gruber can't tell a fully AI-generated article from a human-written one, perhaps he shouldn't care so much. NB: I was told yesterday it's apparently a meme to call out Claude-generated output, but here I am, as…

Gruber really doesn't like any EU regulation

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#200

Post by zimpenfish »

npilk wrote: post #19
syrrim wrote: post #9

> I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo ran…

I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want. On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense. Even good human writers would probably struggle with this constraint. It wou…

> Even good human writers would probably struggle with this constraint.

But that would be a fun writing exercise, I think. Thoroughly in the oulipo wheelhouse.

Maybe generate a Markov chain table over all of Project Gutenberg and then say every 10th word is whatever the Markov Chain thinks it should be at that point?

Or every Nth word has a P% possibility to be constrained by the chain? Optionally with the possibility building for each skipped word to guarantee it happens at some point. Bonus with this approach is that the human can't game the words leading up to the constraint because you don't know when it will happen.

Post reply on HN