Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

daringfireball.net

Post reply on HN
Print view View original

101–110 of 776 posts

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#101

Post by 0x_rs »

I'd encourage reading this paper, and literature on scaling laws in autoregressive models: https://arxiv.org/abs/2303.11156

Total variation distance has been measured to decrease as you scale a model, and that is the primary mechanism "watermarking" as discussed in the Anthropic announcement relies on. It becomes more difficult to reliably detect text as a fixed sample count without tweaking the distribution further. Either way, it's a minor problem that will be addressed over time, compared to the issue of who can detect this without guessing or developing their own sets: providers not releasing a way to detect any such watermarks without going through them makes this entire approach hostile to the public. The EU regulation on this subject is interesting, although again most certainly not the primary driver for these practices:

"1.1.2: Signatories will ensure that AI-generated or manipulated content is marked with an imperceptible watermark, with the exception of very short text. For free-form text longer than 200 tokens, watermarking still needs to be applied, even though it may have lower reliability compared to that of watermarking very long text"

A proper, effective and useful law would have required providers to regularly release datasets to run your own verification on any text released within a fixed interval of time, presumably once out of rotation. Instead, it only talks about exposing an user interface going through their own services:

"Signatories will ensure access to their detection solution through a user interface appropriate for the audience of end-users that may eventually be exposed to the content generated or manipulated by their AI system. [...] Any restriction to the access will be limited in time until more reliable and robust detection mechanisms have emerged and have been adopted as the state of the art for detection mechanisms for the watermarking of free-form text evolves."

Most interestingly, in line with the EU's mass-surveillance program, an alternative solution to watermarking where it may not be sufficient is also suggested, although only optional for now:

"Where appropriate and taking into account potential trade-offs related to privacy and security, as well as scalability challenges and costs, Signatories may implement as an optional supplementary measure fingerprinting or logging solutions for AI-generated or manipulated content which allow for checking whether content has been generated or manipulated by their AI system. For example, direct logging may be appropriate for text content, whereas fingerprinting approaches may be preferable for audio and visual content."

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#102

Post by npilk »

brookst wrote: post #68
npilk wrote: post #19

Earlier quoted context omitted.

I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want. On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense. Even good human writers would probably struggle with this constraint. It wou…

This is the classic misunderstanding that LLMs only pick the next token at a time. Really, they are coalescing the probabilities of a range of tokens at a time. There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”.

But on some level there is uncertainty, right? Even if it’s not token-specific but at the word- or phrase-level? Otherwise what does the temperature setting do? Or has architecture changed significantly in the background?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#104

Post by cubefox »

pibaker wrote: post #58

I think it's pretty dishonest of Anthropic to frame their watermark as EU regulation compliance. The EU regulation, from my understanding, requires AI content to be labeled for human viewers. In the meanwhile the Anthropic new release on the watermark says this. > The difference between watermarked and un-watermarked text will not be distinguishable to readers https://www.anthropic.com/news/claude-text-watermark Whic…

> The EU regulation, from my understanding, requires AI content to be labeled for human viewers.

How would that work? Claude appending " written by AI" to each of its messages? That would both be impractical and useless.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#105

Post by zmmmmm »

smallerize wrote: post #7

Translation: No one can ever again use Claude for proofreading their own prose unless they’re willing to risk that the whole thing might be flagged as having been generated by Claude. I think that was intended, yes.

The question is, when is the "pro writer" version coming that lets you control this behaviour but costs more? Like night follows day, this will happen.

They will need to dodge around the EU requirements but it will probably just come down to an alternative method to watermark or a contractual assurance you won't mis-represent the source of the text.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#106

Post by moralestapia »

brookst wrote: post #68
npilk wrote: post #19

Earlier quoted context omitted.

I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want. On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense. Even good human writers would probably struggle with this constraint. It wou…

This is the classic misunderstanding that LLMs only pick the next token at a time. Really, they are coalescing the probabilities of a range of tokens at a time. There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”.

There are diffusion-based models and transformer-based models (and many other "architectures"), so your comment does not make sense.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#107

Post by demibabs »

dare944 wrote: post #31
ButlerianJihad wrote: post #10

Earlier quoted context omitted.

It is quite just, if you think about it. Human works are copyrighted and protected at the moment of creation. All rights reserved. Yet, LLM outputs are uncopyrightable. Therefore, if Claude or any AI has processed my copyrighted work, the end result is uncopyrightable and in the Public Domain. The public has a right to know: is this a human copyrighted work, an LLM PD work, or is the human falsely claiming authorship…

As I understand it, the current watermarking methods rely on a secret key, making the detection schemes a black box to anyone not in possession of the key. This means organizations like Anthropic are free to make any claim about authorship they want, true or not, and no one can call them on it.

Part of the legislation requires them to make a public AI text detector (ala GPTZero I assume).

Wouldn’t having that be enough to eventually reverse engineer the key?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#108

Post by inigyou »

brookst wrote: post #68
npilk wrote: post #19

Earlier quoted context omitted.

I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want. On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense. Even good human writers would probably struggle with this constraint. It wou…

This is the classic misunderstanding that LLMs only pick the next token at a time. Really, they are coalescing the probabilities of a range of tokens at a time. There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”.

No, they really do one at a time. You're incorrect on that.

Mathematically, a long chain of conditional probabilities is equivalent to a single probability over the whole range. But computationally, for that to work out, the computation for the first probability needs to somehow consider all the downstream probabilities depending on it, which obviously isn't how autoregressive language models work. They can pack in as much downstream computation as their neural architecture allows for, which is quite a lot.

Suppose in some context you have three equally plausible conpletions after "Be": "tween a rock and a hard place", "twixed he stood there" and "lieve he can fly". To model this probability distribution of the whole sentence, the next token "tw" needs to appear at 2/3 probability and "lie" at 1/3. After "tw" would be a 1/2 chance of "ix" and a 1/2 chance of "een"; after "lie" would be a 100% chance of "ve " and in any case the rest of the sentence after that would be 100%.

The model needs to somehow "think ahead" to know those are the possible completions. For example if "lieve he can swim like a dolphin" was another equally plausible completion, that first token would need to be 50/50 instead of 67/33. So the computation of the first token somehow needs to encode the fact that the guy thinks he can fly but not swim, even though it doesn't become relevant in the output until several tokens later.

In practice this probably happens to some degree but definitely doesn't happen perfectly. To perfectly model the first token's probability distribution, it would have to include knowledge of the entire distribution of all possible outputs, which is just not happening. So it approximates. Surprisingly, the approximation is good enough to produce language.

You can see this breaking down in the seahorse emoji incident from last year. When you ask the model if there's a seahorse emoji, it first completes "Yes," as if a few tokens later it's about to produce a seahorse emoji. But when it actually gets to the token that would produce a seahorse emoji, it can't because there isn't one. But it's already outputted "Yes, the seahorse emoji is" and can't just go back and change that to "No, there's no seahorse emoji." Some models would try a few times and then say there isn't one or a system error seems to be making them unable to produce one, other models (including then-current ChatGPT) would loop forever with ensuing hilarity.

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#109

Post by inigyou »

moralestapia wrote: post #106
brookst wrote: post #68

Earlier quoted context omitted.

This is the classic misunderstanding that LLMs only pick the next token at a time. Really, they are coalescing the probabilities of a range of tokens at a time. There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”.

There are diffusion-based models and transformer-based models (and many other "architectures"), so your comment does not make sense.

Are there any diffusion-based or otherwise non-transformer-based models in mainstream use?

Re: Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

#110

Post by inigyou »

disillusioned wrote: post #100
npilk wrote: post #19

Earlier quoted context omitted.

I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want. On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense. Even good human writers would probably struggle with this constraint. It wou…

It's a bit like trying to finish a sentence when you're really stoned... you vaguely remember the preceding couple of words you've said but don't really know how you got there and now you're wandering in the forest trying to stumble on coherency. Well, I suppose it's nearly the opposite of that experience, upon further review. But for some reason, that's where my head jumped.

You're thinking of Markov chains.
Post reply on HN