Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we measured AI writing across arXiv, and where the measurement breaks

unslop.run

Post reply on HN
Print view View original

141–150 of 185 posts

Re: How we measured AI writing across arXiv, and where the measurement breaks

#141

Post by yorwba »

NitpickLawyer wrote: post #84
pbui wrote: post #65

I'm not sure what to think... I uploaded a PyHPC workshop paper I wrote in 2011 and it said 27% machine. I also uploaded my PhD dissertation from 2012 and got back 40% machine, which is just barely below the 42% threshold. I don't publish anymore... but does this mean I wrote like a LLM or did LLMs learn from me? :p Update: I also uploaded a IEEE CLUSTERS paper I wrote in 2015 and it came back 74% machine written :|

When "detectors" first started popping up all over the place, all of them rated the declaration of independence as 100% AI written, so... Yeah, these things just don't work. And what's even more dangerous is that people that don't understand how any of it works use these tools, and accuse people of using AI, sometimes with grave consequences. Students have been through this, at all levels of education.

They use a "threshold calibrated so pre-ChatGPT papers flag at 0.4%", so these things do work most of the time. It also means that there are known false positives, so for any given paper, scoring above the threshold isn't irrefutable proof of AI usage. But for things like estimating the overall proportion of AI writing, you only need to be correct on average, so individual false positives don't matter much.

Re: How we measured AI writing across arXiv, and where the measurement breaks

#142

Post by s1artibartfast »

twoodfin wrote: post #125
s1artibartfast wrote: post #99

Earlier quoted context omitted.

Many people are bad at writing, including scientists. Improvements could be articulating a concept in a way that the reader will understand it clearly.

My hypothesis is that an LLM’s ability to concoct prose that will convince even an expert of the validity of an idea is largely independent of the LLM’s ability to validate the idea itself, or whether the idea is correct in the first place. Especially if it’s being prompted to do the former, not the latter. My anecdotal evidence is the LLM-generated, inchoate technical dross that is routinely upvoted onto the hn fron…

I agree prose and validation are different. I dont know anyone who contests this.

Re: How we measured AI writing across arXiv, and where the measurement breaks

#143

Post by thesmtsolver2 »

NitpickLawyer wrote: post #84
pbui wrote: post #65

I'm not sure what to think... I uploaded a PyHPC workshop paper I wrote in 2011 and it said 27% machine. I also uploaded my PhD dissertation from 2012 and got back 40% machine, which is just barely below the 42% threshold. I don't publish anymore... but does this mean I wrote like a LLM or did LLMs learn from me? :p Update: I also uploaded a IEEE CLUSTERS paper I wrote in 2015 and it came back 74% machine written :|

When "detectors" first started popping up all over the place, all of them rated the declaration of independence as 100% AI written, so... Yeah, these things just don't work. And what's even more dangerous is that people that don't understand how any of it works use these tools, and accuse people of using AI, sometimes with grave consequences. Students have been through this, at all levels of education.

I replied to a comment with a structured informal proof on HN.

A well established user here was adamant that I used AI because apparently humans never ever wrote proofs.

This was a while ago. Any well crafted human output is now being dismissively cast as AI if the reader is challenged by the output intellectually/politically.

Re: How we measured AI writing across arXiv, and where the measurement breaks

#144

Post by jasonfarnon »

yorwba wrote: post #141
NitpickLawyer wrote: post #84

Earlier quoted context omitted.

When "detectors" first started popping up all over the place, all of them rated the declaration of independence as 100% AI written, so... Yeah, these things just don't work. And what's even more dangerous is that people that don't understand how any of it works use these tools, and accuse people of using AI, sometimes with grave consequences. Students have been through this, at all levels of education.

They use a "threshold calibrated so pre-ChatGPT papers flag at 0.4%", so these things do work most of the time. It also means that there are known false positives, so for any given paper, scoring above the threshold isn't irrefutable proof of AI usage. But for things like estimating the overall proportion of AI writing, you only need to be correct on average, so individual false positives don't matter much.

It would be interesting to see how that .4% varies across fields on arXiv. I imagine some areas influenced LLM writing much more than others and are more susceptible to false positives. Anyway it's a little strange to me that these detectors do so poorly (at least by what I hear on this site). There is as much labeled data as you would need to train on pre-LLM human vs LLM text. If their out of sample errors are as good as has been claimed, the only methodological flaw I can think of is that post-LLM human text and LLM text have been converging.

Re: How we measured AI writing across arXiv, and where the measurement breaks

#145

Post by nullc »

What's the point of looking when no one even cares?

consider the fraudster that went around suing people on the basis of his absurd claims of being bitcoin's creator. He's now transitioned to using AI to gather graduate degrees and is obtaining masters and doctoral degrees at a regular place and writing multiple 'papers' per day that are all quite obviously AI slop.

People report his cheating and publications and simply no one cares... and this is someone court adjudicated to have fabricated evidence in court on a massive scale, including through the use of AI.

But when it comes to the degrees and publication everyone involved that wanted paid got paid, and apparently that's all that matters.

Re: How we measured AI writing across arXiv, and where the measurement breaks

#146

Post by mewpmewp2 »

jknoepfler wrote: post #97
ryandvm wrote: post #85

There some real game theory mechanics at play when it comes to LLM usage in corporations. Devs are cranking out superficially superior code and documentation by just aiming the Claude Code fire hose at everything they can. Leadership encourages this because from what they can tell, there is no downside. It's hard to say if this code is structurally better or worse than before, but it's certainly voluminous and as far…

> "superficially superior code" Can you unpack that a bit? It produces measurably, meaningfully inferior code everywhere I see it in use. > "leadership encourages this because from what they can tell, there is no downside" As said 'leadership' I find this a bit puzzling. I'm seeing strong, quantifiable evidence of increasing churn, increasing incident count, and length of downtime from the date of our biggest push in…

What size company are you talking about?

Re: How we measured AI writing across arXiv, and where the measurement breaks

#147

Post by sean_pedersen »

pbui wrote: post #65

I'm not sure what to think... I uploaded a PyHPC workshop paper I wrote in 2011 and it said 27% machine. I also uploaded my PhD dissertation from 2012 and got back 40% machine, which is just barely below the 42% threshold. I don't publish anymore... but does this mean I wrote like a LLM or did LLMs learn from me? :p Update: I also uploaded a IEEE CLUSTERS paper I wrote in 2015 and it came back 74% machine written :|

"Any attempt to build AI generated content (deep fake) detection systems is flawed, since the outputs of such a system may be used to train an even better fake data generator. This leads to an equilibrium state of digital uncertainty: nothing in the digital realm can be deemed as real anymore - only as digital. I do not care if a digital artifact is human or AI made - I only care if it is useful to me. Useful content is on point, factual and at best surprising (teaches something new)." - https://seanpedersen.github.io/posts/digital-uncertainty/

Re: How we measured AI writing across arXiv, and where the measurement breaks

#148

Post by adamgordonbell »

JamesBarney wrote: post #75
paxys wrote: post #67

Earlier quoted context omitted.

What detector are you using? How can you be sure of its accuracy given that every commercial AI detector has been debunked?

Almost every. Pangram is pretty accurate on longer texts. I haven't seen any glaring examples of false positives.

This! Pangram is very good. They claim 1 in 10,000 fp rate.

I had to change my mind on AI detectors after playing around with it.

It would be interesting to hear how this detector compares. It also seems to be aiming for low fp rate.

Re: How we measured AI writing across arXiv, and where the measurement breaks

#149

Post by roysting »

thesmtsolver2 wrote: post #143
NitpickLawyer wrote: post #84

Earlier quoted context omitted.

When "detectors" first started popping up all over the place, all of them rated the declaration of independence as 100% AI written, so... Yeah, these things just don't work. And what's even more dangerous is that people that don't understand how any of it works use these tools, and accuse people of using AI, sometimes with grave consequences. Students have been through this, at all levels of education.

I replied to a comment with a structured informal proof on HN. A well established user here was adamant that I used AI because apparently humans never ever wrote proofs. This was a while ago. Any well crafted human output is now being dismissively cast as AI if the reader is challenged by the output intellectually/politically.

I’ve been accused of being a bot myself. I’ve started leaving some incorrect autocorrects in, hopping it would stave off people for whom it seems making accusations of being a bot is soothing or a coping mechanism when they’re confronted with information that does not fit into their world view or perception of self.

Re: How we measured AI writing across arXiv, and where the measurement breaks

#150

Post by rpm91 »

I tried the detector on several of my own Stack Overflow answers, and the fourth or fifth one I tried was flagged as "85 % machine / likely machine-written" with the default settings, well above the default sensitivity threshold of 42%. When I turn off the option to strip LaTeX commands (the source is Markdown), that jumps to 96% machine.

Admittedly, a Stack Overflow answer is somewhat outside the realm of scientific writing, so it's still possible that the detector may be accurate within that domain. That said, it's a cautionary tale on the hazards of applying classifiers like this outside of the domain that they were trained on.

Post reply on HN