Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I wouldn't say Pangram is broken, but I would say that it's brittle

freddiedeboer.substack.com

Post reply on HN
Print view View original

41–45 of 45 posts

Re: I wouldn't say Pangram is broken, but I would say that it's brittle

#41

Post by runako »

Smaug123 wrote: post #38
runako wrote: post #32

This product does not even have a plausible theory of how it could work. LLM-generated text does not carry a watermark or other identifying marks. The "theory" is that an LLM trained on human writing, to mimic human writing, can be distinguished from actual human writing in under 100 words. Notably the first diagram on the research overview page ( https://www.pangram.com/research/how-it-works ) shows feedback for "mi…

Claude, a general-purpose model, can identify me, personally with stylometry in about 200 words. Is it really such a stretch to believe it’s possible for a special-purpose model to identify the ten or so main LLMs crossed with the fifty or so main styles people gave them write in?

> Claude, a general-purpose model, can identify me, personally

It cannot. this is a misconception. A human being is fully capable of writing 200 words that Claude will identify as not being written by them, because human beings are far more complex than Claude. A human can even choose to deliberately write in the style of a different human, even one who does not exist.

Sometimes people are writing instruction manuals; those are not written like their professional emails, which are not written like their personal emails. It is normal for people to be able to write in different voices/styles/etc. People code switch, people write for different audiences, people change over time, people are hurried or tired or sick, etc.

So no, an LLM cannot identify you uniquely in 200 words. But more to the point, most human communication is not in training sets. And Pangram has no way of course-correcting on the vast amount of data that is not in its training sets.

By comparison: the autonomous vehicle companies actually do need their products to verifiably work. So they also feed back human-analyzed data from real trips into their models. They can tell the model where it was right or wrong in the real world. This is the part Pangram cannot do! Pangram deployed at a university may be used to accuse a student of cheating, but then Pangram will never know for sure whether the text in question was written by a human or machine. The feedback loop is missing a critical step!

Re: I wouldn't say Pangram is broken, but I would say that it's brittle

#42

Post by Smaug123 »

runako wrote: post #41
Smaug123 wrote: post #38

Earlier quoted context omitted.

Claude, a general-purpose model, can identify me, personally with stylometry in about 200 words. Is it really such a stretch to believe it’s possible for a special-purpose model to identify the ten or so main LLMs crossed with the fifty or so main styles people gave them write in?

> Claude, a general-purpose model, can identify me, personally It cannot. this is a misconception. A human being is fully capable of writing 200 words that Claude will identify as not being written by them, because human beings are far more complex than Claude. A human can even choose to deliberately write in the style of a different human, even one who does not exist. Sometimes people are writing instruction manuals…

I have run the experiment like seven times now on different tracts of text, given to people who are not me. It’s a point of simple fact that Opus 4.7 can identify me when I’m writing fresh text in my voice. Does that change your conclusion if you were to grant it for the sake of argument (notwithstanding the fact that it’s actually true)? Or is your objection “it can identify one of your voices, the most commonly used one, and not the others” or something like that?

Re: I wouldn't say Pangram is broken, but I would say that it's brittle

#43

Post by runako »

Smaug123 wrote: post #42
runako wrote: post #41

Earlier quoted context omitted.

> Claude, a general-purpose model, can identify me, personally It cannot. this is a misconception. A human being is fully capable of writing 200 words that Claude will identify as not being written by them, because human beings are far more complex than Claude. A human can even choose to deliberately write in the style of a different human, even one who does not exist. Sometimes people are writing instruction manuals…

I have run the experiment like seven times now on different tracts of text, given to people who are not me. It’s a point of simple fact that Opus 4.7 can identify me when I’m writing fresh text in my voice. Does that change your conclusion if you were to grant it for the sake of argument (notwithstanding the fact that it’s actually true)? Or is your objection “it can identify one of your voices, the most commonly use…

Correct -- it can identify one of your (current) voices, written with your knowledge that the text will be used for the purpose of having an LLM determine whether you wrote it. This is a mentalist setup.

Expanding on that...you are fully capable of writing e.g. product instructions, or marketing copy, or religious verse, or a poem, or a fictional quote from a fictional character in your upcoming novel. I would consider it unlikely that any LLM could identify you as the writer of any of those (though Claude might infer it was you given your chat history). The point is that Pangram claims to be able to do exactly that!

Pangram is also making the claim that they can, to a high degree of certainty, identify when someone else wrote text and claimed it to be yours. This, when Pangram has not ever seen a writing sample of either writer. This is quite obviously ridiculous!

Re: I wouldn't say Pangram is broken, but I would say that it's brittle

#44

Post by Smaug123 »

runako wrote: post #43
Smaug123 wrote: post #42

Earlier quoted context omitted.

I have run the experiment like seven times now on different tracts of text, given to people who are not me. It’s a point of simple fact that Opus 4.7 can identify me when I’m writing fresh text in my voice. Does that change your conclusion if you were to grant it for the sake of argument (notwithstanding the fact that it’s actually true)? Or is your objection “it can identify one of your voices, the most commonly use…

Correct -- it can identify one of your (current) voices, written with your knowledge that the text will be used for the purpose of having an LLM determine whether you wrote it. This is a mentalist setup. Expanding on that...you are fully capable of writing e.g. product instructions, or marketing copy, or religious verse, or a poem, or a fictional quote from a fictional character in your upcoming novel. I would consid…

> written with your knowledge that the text will be used for the purpose of having an LLM determine whether you wrote it

Actually no. I've done the experiment under those conditions, of course, but I've also out of mere interest just shoved some text in after the fact (most recently, to see whether Claude would de-anonymise the first two paragraphs of my answer to a questionnaire out of the box - it did).

"Quite obviously ridiculous" I simply disagree with. My compendium of Sherlock Holmes stories came with a book at the end, "The Casebook of Sherlock Holmes", which was century-old fanfiction, containing stories written by people trying to write Arthur Conan Doyle writing Sherlock Holmes, and it was clearly not him. I really think you massively underestimate just how many bits of information are contained in a paragraph of text!

Re: I wouldn't say Pangram is broken, but I would say that it's brittle

#45

Post by runako »

Smaug123 wrote: post #44
runako wrote: post #43

Earlier quoted context omitted.

Correct -- it can identify one of your (current) voices, written with your knowledge that the text will be used for the purpose of having an LLM determine whether you wrote it. This is a mentalist setup. Expanding on that...you are fully capable of writing e.g. product instructions, or marketing copy, or religious verse, or a poem, or a fictional quote from a fictional character in your upcoming novel. I would consid…

> written with your knowledge that the text will be used for the purpose of having an LLM determine whether you wrote it Actually no. I've done the experiment under those conditions, of course, but I've also out of mere interest just shoved some text in after the fact (most recently, to see whether Claude would de-anonymise the first two paragraphs of my answer to a questionnaire out of the box - it did). "Quite obvi…

The Holmes example is off-base as you had quite obviously been exposed to Holmes (e.g. it was in your training data). Pangram is asserting they don't need that to identify an individual. An individual who may still be learning and developing their writing style/voice, etc.

Put this in the mentalist bucket. Sure, for parlor tricks like you describe, it can be entertaining because the stakes are low and we accept the error rate. Should you discipline a student solely on the basis of Pangram output (which is happening in the real world)? Absolutely not.

Post reply on HN