> Here is the method, the results, and an honest account of the limitations.
This paper was written using AI, to be honest
How we measured AI writing across arXiv, and where the measurement breaks
161–170 of 185 posts
Re: How we measured AI writing across arXiv, and where the measurement breaks
#162Earlier quoted context omitted.
Which part is a sin? Using LLMs to deal with a lack of English-language fluency? I am a scientist (actually a mathematician, if it matters), and, if that's the way to deal with the practical hegemony of English in the scientific literature, then I have no problem with it. Rather that than people with important ideas can't get them before the scientific community. As long as the authors personally check and stand behi…
https://lists.isocpp.org/std-proposals/att-0486/Reply_to_Zer... Here's a paper by a non native English speaker. The lack of the formal style doesn't cause any issue. Preciseness is what matters.
To your tastes maybe. Will it get through the editor's desk at a scientific journal as easily?
Re: How we measured AI writing across arXiv, and where the measurement breaks
#163I once read a comment on hn that says that ai detection in writing is so hard because text doesn't hold enough information.
Now I'm no researcher,but that intuitivly makes sense to me. I mean, apart from the obvious cases, how can you detect ai written text?
Re: How we measured AI writing across arXiv, and where the measurement breaks
#164I've looked at this same problem in an academic environment and have come to conclude that there is no way to reliably detect AI writing using only text. The reason for this is that no detector can take two identical inputs and classify one as synthetic (LLM) and one as organic (Human) and this situation can easily happen at the sentence or even paragraph level. Posed as a question, if a human writes a paragraph that…
When some closed models are "retired", interestingly we might not even be able to extract all their tells. We'd be limited to people writing about them on forums (where they don't often specify version, and there's a lot of memes and confirmation bias) and I don't know, doing some statistical Bayesian guessing that given the likelihood it's FooGPT 10.5, maybe we should update our beliefs about its characteristics etc.
Re: How we measured AI writing across arXiv, and where the measurement breaks
#165[flagged]
Re: How we measured AI writing across arXiv, and where the measurement breaks
#166[flagged]
Re: How we measured AI writing across arXiv, and where the measurement breaks
#167> We took papers submitted in 2021 and 2022, before ChatGPT, treated them as ground-truth human
How is the author so sure that people or companies weren’t already using early tools at that time?
Re: How we measured AI writing across arXiv, and where the measurement breaks
#168Earlier quoted context omitted.
[flagged]
Which part is a sin? Using LLMs to deal with a lack of English-language fluency? I am a scientist (actually a mathematician, if it matters), and, if that's the way to deal with the practical hegemony of English in the scientific literature, then I have no problem with it. Rather that than people with important ideas can't get them before the scientific community. As long as the authors personally check and stand behi…
It would be nice if scientists could write and publish in their first language. If a machine translation brings it to a wider audience fine, but the lack of disclosure of who authored the paper is a problem.
Re: How we measured AI writing across arXiv, and where the measurement breaks
#169Earlier quoted context omitted.
This is fallacious. The fact that someone used LLM to write their paper does not negate its findings.
That's not the point though, the point being made ist that an LLM written paper very likely doesn't actually find something despite looking like it does on the surface
I would like some sort of proof of this claim. I genuinely don't know, but I would suspect that the gap between "things found" in LLM v. Human papers is not as great as we would like it to be.
Re: How we measured AI writing across arXiv, and where the measurement breaks
#170Earlier quoted context omitted.
Oh, to be technically correct: AI hallucinates and makes up stuff 100% percent of the time. Never been a fan of that word for this. Again, I fail to see the problem here that isn't solved by careful reading WHICH IS WHAT PEOPLE SHOULD BE DOING ANYWAY. I would like to see room for AI disclosure, maybe a statement of "this is how much AI I used." But this blanket X% of this looks like AI? Again, so what?
If as a primary school student you had needed to reverify every fact /experiment that was presented in your textbooks, you would not have gotten very far. Trust is very important to human progress.
Absolutely. Which is why what I think I'm saying is that -- it's not ncessarily important who or what WROTE the paper, it's simply important that human-in-the-loop is strong.