Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do systems (or people) detect when a text is written by an LLM

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 72 posts

Re: Ask HN: How do systems (or people) detect when a text is written by an LLM

#21

Post by derwiki »

alex43578 wrote: post #11
Someone1234 wrote: post #4

They cannot. Unfortunately many believe they can, and it is impossible to disprove. So now real people need to write avoiding certain styles, because a lot of other people have decided those are "LLM clues." Bullets, EM Dash, certain common English phases or words (e.g. Delve, Vibrant, Additionally, etc)[0]. Basicaly you need to sprinkle subtle mistakes, or lower the quality of your written communications to avoid ac…

Someone with native fluency in American English can (should) be able to tell the difference between human writing and unpolished AI copy-paste. Essentially 0 people use emoji to create a bulleted list. Nobody unintentionally cites fake legal precedents or non-existent events, articles, or papers. Even the “it’s not X, it’s Y” structure, in the presence of other suspicious style/tone cues signals LLM text.

Emojis for lists: completely agree with you, but presumably this was learned in training?

Re: Ask HN: How do systems (or people) detect when a text is written by an LLM

#23

Post by sheepscreek »

Someone1234 wrote: post #4

They cannot. Unfortunately many believe they can, and it is impossible to disprove. So now real people need to write avoiding certain styles, because a lot of other people have decided those are "LLM clues." Bullets, EM Dash, certain common English phases or words (e.g. Delve, Vibrant, Additionally, etc)[0]. Basicaly you need to sprinkle subtle mistakes, or lower the quality of your written communications to avoid ac…

This is the correct answer. We’re at a point where it will soon be safer to assume a human or someone with agency and their approval wrote the text, than to completely dismiss it as “written by LLM” or a human.

So judge the content on its merit irrespective of its source.

Re: Ask HN: How do systems (or people) detect when a text is written by an LLM

#24

Post by spindump8930 »

moonu wrote: post #3

Pangram is probably the best known example of a detector with low false positives, they have a research paper here: https://arxiv.org/pdf/2402.14873 . They do have an API but not sure if you need to request access for it. For humans I think it just comes down to interacting with LLMs enough to realize their quirks, but that's not really fool-proof.

Pangram has time after time been shown as the only detector that mostly works. And that paper is pretty old now! There are recent papers from academics independently bench-marking and studying detectors e.g. https://arxiv.org/abs/2501.15654

Re: Ask HN: How do systems (or people) detect when a text is written by an LLM

#27

Post by mulr00ney »

Someone1234 wrote: post #4

They cannot. Unfortunately many believe they can, and it is impossible to disprove. So now real people need to write avoiding certain styles, because a lot of other people have decided those are "LLM clues." Bullets, EM Dash, certain common English phases or words (e.g. Delve, Vibrant, Additionally, etc)[0]. Basicaly you need to sprinkle subtle mistakes, or lower the quality of your written communications to avoid ac…

> Unfortunately many believe they can, and it is impossible to disprove. So now real people need to write avoiding certain styles, because a lot of other people have decided those are "LLM clues." Bullets, EM Dash, certain common English phases or words (e.g. Delve, Vibrant, Additionally, etc)[0].

I think people will be able to detect the lowest-user-effort version of LLM text pretty reliably after a while (ie what you describe; many people have a good sense of LLM clues). But there's probably a *ton* of LLM text out there where some of the instructions given were "throw a few errors in", "don't use bullet points or em dashes", "don't do the `it's not this, it's that` thing" going undetected.

And then those changes will get built into ChatGPT's main instructions, and in a few months people will start to pick up on other indicators, and then slightly smarter/more motivated users will give new instructions to hide their LLM usage... (or everyone stops caring, which is an outcome I find hard to wrap my head around)

Re: Ask HN: How do systems (or people) detect when a text is written by an LLM

#28

Post by Joel_Mckay »

Someone1234 wrote: post #4

They cannot. Unfortunately many believe they can, and it is impossible to disprove. So now real people need to write avoiding certain styles, because a lot of other people have decided those are "LLM clues." Bullets, EM Dash, certain common English phases or words (e.g. Delve, Vibrant, Additionally, etc)[0]. Basicaly you need to sprinkle subtle mistakes, or lower the quality of your written communications to avoid ac…

Indeed, isomorphic plagiarism by its nature forms strong vector search paths that were made from stealing both global websites, real peoples work, and LLM user-base input/markdown.

However, reasoning models adding a random typo to seem less automated, still do not hide the fairly repeatable quantized artifacts from the training process. For LLM, it is rather trivial to find where people originally scraped the data from if they still have annotated training metadata.

Finally, reading LLM output is usually clear once one abandons the trap of thinking "I think the author meant [this/that]", and recognizing a works tone reads like a fake author had a stroke [0]. =3

[0] https://en.wikipedia.org/wiki/Stroke

Re: Ask HN: How do systems (or people) detect when a text is written by an LLM

#30

Post by roncesvalles »

prmph wrote: post #19
alex43578 wrote: post #11

Earlier quoted context omitted.

Someone with native fluency in American English can (should) be able to tell the difference between human writing and unpolished AI copy-paste. Essentially 0 people use emoji to create a bulleted list. Nobody unintentionally cites fake legal precedents or non-existent events, articles, or papers. Even the “it’s not X, it’s Y” structure, in the presence of other suspicious style/tone cues signals LLM text.

Also one big tell that is hard to hide is making verbose lists with fluff but little actual informative content. Ask an LLM to read your project specs and add a section headed: Performance Optimizations, to see an example of this Another is a certain punchy and sensationalist style that does not change throughout a longer piece of writing.

Exactly, it's the monotony of the style that gives it away.
Post reply on HN