Wow, OpenAI pulling the plug on its AI Classifier tool really shows how tough it is to spot the difference between human and AI-generated text. It's a bummer that it didn't work out as planned, but hey, that's how we learn, right?
OpenAI shuts down its AI Classifier due to poor accuracy
241–250 of 292 posts
Re: OpenAI shuts down its AI Classifier due to poor accuracy
#242No official announcement is lame
Re: OpenAI shuts down its AI Classifier due to poor accuracy
#243Earlier quoted context omitted.
Eh, I'd consider that a failure of employee training and reverse the situation by giving out a weekly bonus to shifts that did not fail to put the security stickers on. Kinda like if they forgot to put the security seal on your aspirin, I'm not going to take them all off because someone forgot to run production with all the bottles sealed.
The bottle of Aspirin goes through many hands between the manufacturer and you including sitting on an unattended shelf open to the public. The person making the pizza is working for the same company as the person delivering it, or may even be the same person. If you can't trust the pizza co delivery person then you probably shouldn't trust the person making it either.
Nowadays food delivery often goes through third party. Eg Uber eats, Foodora, Bolt, Grab, etc.
Re: OpenAI shuts down its AI Classifier due to poor accuracy
#244Earlier quoted context omitted.
Why even care if it is written by a machine or not? I am not sure it matters as much as people think.
Well, I teach English as a second language in a non-English speaking country. I often used short essays and diary-writing for homework. The students have had lots of English input over the years, but not much experience with output. So, writing assignments work out very well for them. Alas, with ChatGPT on the rise here, they no longer have to write it themselves. The upshot of which is, the useful writing assignment…
What if they accompany the writings with a recording of them reading it aloud?
I think you could pick up right quick on who understood what they wrote, and who didn't.
Re: OpenAI shuts down its AI Classifier due to poor accuracy
#245Earlier quoted context omitted.
Flagged "human" but actually "LLM" is not a false positive, but a false negative.
It depends how the question is framed: are you asking to confirm humanity, or confirm LLM. If you are asking, is this LLM text Human generated, and it says Human (yes), then it is false positive. If you are asking is this LLM generated text LLM generated, and is says and it says Human (no), then it is a false negative.
That seems like a restrictive binary. Are there not other entities which generate text? What if a gorilla uses ASL that is transcribed? ELIZA could generate text, after a fashion, as a precursor to LLM. It seems like there's a number of automated processes that could take data and generate text, sort of, like weather reports, no?
So I think the only thing a mythical detector could determine would be LLM, or non-LLM, and let us take it from there. But detectors are bunk; I've had first-hand experience with that.
Re: OpenAI shuts down its AI Classifier due to poor accuracy
#246If we can't tell what is written by AI, how will we know if AI is just stuck in a feedback loop in which it references itself (and is wrong)?
Re: OpenAI shuts down its AI Classifier due to poor accuracy
#247Earlier quoted context omitted.
Well, I teach English as a second language in a non-English speaking country. I often used short essays and diary-writing for homework. The students have had lots of English input over the years, but not much experience with output. So, writing assignments work out very well for them. Alas, with ChatGPT on the rise here, they no longer have to write it themselves. The upshot of which is, the useful writing assignment…
Is your objective that your students learn, or to police them? In the latter case, yes, you have those two options you mentioned. In the former case, you can just continue as you were. Some will cheat and some will not. The ones who do are only cheating themselves.
My goal is for them to learn. And yes, I can just carry on, and some will cheat. I've caught cheaters in the past, of course, but they were far and few between. With ChatGPT and even improved translators like DeepL, it's hard to get them to do the practice they need to learn.
And as a teacher who really WANTS them to learn and to get that feeling, "Hey, I can actually do this!", it's depressing to think of the one who do cheat themselves. Oh well...
Re: OpenAI shuts down its AI Classifier due to poor accuracy
#248Earlier quoted context omitted.
The whole problem with AI is that it's able to copy some of the superficial indicators of quality content while feeding you lies. You cannot detect quality content without detecting truthfulness. Any heuristic you use in place of that can be copied without actually providing value (which is exactly what ChatGPT does now, when it gets things wrong)
That's the whole problem with LLMs in general. They are designed to be convincing, not necessarily accurate.
thats my take on all this LLM hype: they're great at creative work where accuracy is not important, and assisting in technical work where the user is already able to discern an answer that is accurate from one that is slightly to fully bullshit.
even if an LLM can give an amazing and correct answer 7/10 times, it still takes a human expert to cherrypick which 7 answers are amazing and which are just convincingly-assembled bs.
Re: OpenAI shuts down its AI Classifier due to poor accuracy
#249Earlier quoted context omitted.
Well, I teach English as a second language in a non-English speaking country. I often used short essays and diary-writing for homework. The students have had lots of English input over the years, but not much experience with output. So, writing assignments work out very well for them. Alas, with ChatGPT on the rise here, they no longer have to write it themselves. The upshot of which is, the useful writing assignment…
What if they accompany the writings with a recording of them reading it aloud? I think you could pick up right quick on who understood what they wrote, and who didn't.
That's actually a pretty good idea. I might have to try it selectively, though it'd be too much for 200 students submitting diaries and summaries every week. laugh
They once said I should join Line then we can all talk, then I asked if it's possible to talk in groups of 200+ and their eyes got really big.
Re: OpenAI shuts down its AI Classifier due to poor accuracy
#250We shattered the turing test and now we want to put it back into pandora’s box because we don’t like the repercussions.
“We shattered the Turing test”
The Turing test is typically an interactive back-and-forth, no? Detecting if paragraphs of pre-written LLM output and giving a yes-or-no answer is not the same thing.