Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Claude marks AI-generated content

support.claude.com

Post reply on HN
Print view View original

371–380 of 446 posts

Re: How Claude marks AI-generated content

#371

Post by xena »

andai wrote: post #331
jrflo wrote: post #300

Earlier quoted context omitted.

I think that false positives are inevitable due to the method of watermarking being embedded in the text itself. The output is intended to mimic human writing, therefore it's entirely conceivable that a human could by chance write text that contains the watermark. The odds may be extremely small, but it's not something you could ever guarantee.

I keep hearing how humans are thinking and writing more and more like AI. I think in this case I think it's some kind of cryptographic signature smeared across the token IDs, so I don't think the risk is very high.

You're absolutely right! Humans have been slowly thinking and writing more and more like AI. As people get more and more exposed to the stochastic patterns of large language model tools, it's normal for them to emulate the styles of communication they are exposed to. This is commonly called "brainrot" by those in Gen Z and younger cohorts.

If you find yourself getting to be afflicted by this "brainrot", be sure to go outside and take a moment to ponder what's around you. The grass is there and will be there long after we are all gone. Consider this for a moment as your organic thought processing unit starts to slowly munch away at its internal context window.

Re: How Claude marks AI-generated content

#374

Post by teravor »

wrsh07 wrote: post #221
simonw wrote: post #15

> When a supported Claude model generates text, it weaves an imperceptible watermark directly into the text itself. You won’t see it, and it doesn’t change the meaning, quality, or readability of Claude’s response. I'd like to know a lot more about how that works. A lot of my interactions with Claude return pretty precise text. If I ask it to edit a project and refactor a specific function in several places I know ex…

Scott Aaronson talks about his project at OpenAI to do this^ You can carefully select which pseudorandom number generator (prng) you use to be able to id text of a certain length. I expect there is some performance characteristic you have to manage since you're doing this on every inference, but once you do that it doesn't change the output in any meaningful way (the prng is still a statistically valid prng, it just…

this would be a very heavy watermark application.

there are many simpler methods, for example you can have a tiny windowed transformer operating on the output text and all you do is alter certain words (that don't change meanings) to maximize its surprise. the tiny language model will have a special training regime to build up a somewhat unique view of the language.

we are talking about a 0.5 bit watermark here (existence). I would have zero confidence in being able to reliably remove such a watermark from pretty much any medium.

Re: How Claude marks AI-generated content

#375

Post by dragonwriter »

Dilettante_ wrote: post #120

I notice the "Limitations" section talks about how content only at some point touched by Claude may return a positive, and content that returns a negative may still be Claude generated. But I really would have liked for them to state explicitly that entirely false positives where a piece is fully human-written may still be marked as generated, because too many institutions with the power to ruin someone's life over t…

> But I really would have liked for them to state explicitly that entirely false positives where a piece is fully human-written may still be marked as generated, because too many institutions with the power to ruin someone's life over that have trouble understanding the concept.

This is marketing material aimed, in part, at encouraging the usage you are concerned about, which is why they do not highlight that problem.

Re: How Claude marks AI-generated content

#376

Post by dragonwriter »

dbqpdb wrote: post #369
ed_elliott_asc wrote: post #347

Earlier quoted context omitted.

It’s worse than that, false positives are possible but someone generating text should be able to get ai to change some words and formatting to break the watermarking, then ai detectors can tell them how well they did. I don’t know what the answer but I absolutely know it isn’t this.

I think we need a chain of custody system for content, but that would require browsers, software, websites, operating systems, phones, camera manufacturers, etc to all get on board. But each intermediary or source (optionally) cryptographicaly signs a piece of content that it either generates, edits, or passes along, and the end result at a destination, is that content is either 'trusted' if its cryptographic chain i…

> I think we need a chain of custody system for content, but that would require browsers, software, websites, operating systems, phones, camera manufacturers, etc to all get on board.

It would also require the individual humans you are trying to control to get on board otherwise the analog hole breaks the chain, absent mindboggling levels of physical surveillance on top of the the total monitoring of all electronic data flows that this idea requires.

Re: How Claude marks AI-generated content

#378

Post by 0x_rs »

mucha wrote: post #372
someguynamedq wrote: post #243

Watermarking text is impossible and a fool's errand

I'd recommend that you gather your thoughts and publish a response to this paper : https://arxiv.org/abs/2301.10226

If we go by "fool's errand" as "needless or profitless endeavor", https://arxiv.org/abs/2303.11156 may already be a good enough answer to the paper you cited, so their work is already laid out for them. The green token idea is thoroughly attacked with much more effective techniques than those in the original paper through "recursive paraphrasing". Among some hypotheses in the paper, one is particularly interesting:

>These experiments provide empirical evidence that more advanced LLMs can lead to smaller TV distances. Thus, based on Theorem 1, reliable AI text detection would become increasingly difficult

Re: How Claude marks AI-generated content

#379

Post by jrflo »

andai wrote: post #331
jrflo wrote: post #300

Earlier quoted context omitted.

I think that false positives are inevitable due to the method of watermarking being embedded in the text itself. The output is intended to mimic human writing, therefore it's entirely conceivable that a human could by chance write text that contains the watermark. The odds may be extremely small, but it's not something you could ever guarantee.

I keep hearing how humans are thinking and writing more and more like AI. I think in this case I think it's some kind of cryptographic signature smeared across the token IDs, so I don't think the risk is very high.

I read the original paper they're basing this off of and I think you're right. I do wonder how much of a quality tradeoff there is with perturbing the next token probability distribution. My intuition tells me that a more "prominent" watermark will necessarily degrade output quality. If they are trying to balance quality and watermark prominence, I wonder if that affects the FPR.

Re: How Claude marks AI-generated content

#380

Post by wrsh07 »

EE84M3i wrote: post #232
wrsh07 wrote: post #221

Earlier quoted context omitted.

Scott Aaronson talks about his project at OpenAI to do this^ You can carefully select which pseudorandom number generator (prng) you use to be able to id text of a certain length. I expect there is some performance characteristic you have to manage since you're doing this on every inference, but once you do that it doesn't change the output in any meaningful way (the prng is still a statistically valid prng, it just…

https://scottaaronson.blog/?p=6823 is the source?

That's what I excerpted, although I had seen it presented from his talk at Stony Brook
Post reply on HN