Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI shuts down its AI Classifier due to poor accuracy

decrypt.co

Post reply on HN
Print view View original

161–170 of 292 posts

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#161

Post by constantcrying »

siglesias wrote: post #110
constantcrying wrote: post #64

Earlier quoted context omitted.

Even the idea of it is bad, ChatGPT is supposed to write indistinguishably from a human. The "detector" has extremely little information and the only somewhat reasonable criteria are things like style, where ChatGPT certainly has a particular, but by no means unique writing style. And as it gets better it will (by definition) be better at writing in more varied styles.

I'd challenge this assumption. ChatGPT is supposed to convey information and answer questions in a manner that is intelligible to humans. It doesn't mean it should write indistinguishably from humans. It has a certain manner of prose that (to me) is distinctive and, for lack of a better descriptor, silkier, more anodyne, than most human writing. It should only attempt a distinct style if prompted to.

ChatGPT is explicitly trained on human writing it's training goal is explicitly to emulate human writing.

>It should only attempt a distinct style if prompted to.

There is no such thing as an indistinct style. Any particular style it could have would be made distinct by it being the style ChatGPT chooses to answer in.

The answers that ChatGPT gives are usually written in a style combining somewhat dry academic prose and the type of writing you might find in a Public Relations statement. ChatGPT sounds very confident in the responses it generates to the queries of users, even if the actual content of the information is quite doubtful. With some attention to detail I believe that it is quite possible for humans to emulate that style, further I believe that the style was designed by the creators of ChatGPT to make the output of the machine learning algorithm seem more trustworthy.

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#162

Post by doliveira »

sebzim4500 wrote: post #96
zarzavat wrote: post #93

Earlier quoted context omitted.

There’s a much more effective way: store hashes of each output paragraph (with a minimum entropy) that has ever been generated, and allow people to enter a block of text to search the database. It wouldn’t beat determined users but it would at least catch the unaware.

Changing every 10th word defeats that strategy but doesn't defeat a cryptographic bias. Also the cost of storing every paragraph hash might eventually add up even if at the moment it would be negligable compared to the generation cost.

They literally already store the whole conversation...

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#163

Post by frumper »

dontreact wrote: post #156
a13o wrote: post #55

Earlier quoted context omitted.

This is a strawman. First, the AI detection algorithms can't offer anything close to 99.9%. Second, your scenario doesn't analyze another human and issue judgement, as the AI detection algorithms do. When a human is miscategorized as a bot, they could find themselves in front of academic fraud boards, skipped over by recruiters, placed in the spam folder, etc.

It's not a strawman. There are many fundamentally unpredictable things where we can't make the benchmark be 100% accuracy. To make it more concrete on work I am very familiar with: breast cancer screening. If you had a model that outperformed human radiologists at predicting whether there is pathology confirmed cancer within 1 year, but the accuracy was not 100%, would you want to use that model or not?

It's a strawman because they aren't comparable to AI detection tests. A screening coming back as possible cancer will lead to follow up tests to confirm, or rule out. An AI detection test coming back as positive can't be refuted or further tested with any level of accuracy. It's a completely unverifiable test with a low accuracy.

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#164

Post by wmeredith »

ribosometronome wrote: post #153
jedberg wrote: post #151

Latest I heard is that teachers are requiring homework to be turned in in Google Docs so that they can look at the revision history and see if you wrote the whole thing or just dumped a fully formed essay into GDocs and then edited it. Of course the smart student will easily figure out a way to stream the GPT output into Google Docs, perhaps jumping around to make "edits". A clever and unethical student is pretty muc…

Retyping the essay from the chatgpt while actively rewording the occassional sentence seems like it would do it.

It seems like that's nearing the sweet spot of fraud prevention, where committing the act of fraud is as much work as doing the real thing.

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#165

Post by frumper »

wmeredith wrote: post #164
ribosometronome wrote: post #153

Earlier quoted context omitted.

Retyping the essay from the chatgpt while actively rewording the occassional sentence seems like it would do it.

It seems like that's nearing the sweet spot of fraud prevention, where committing the act of fraud is as much work as doing the real thing.

It sounds a lot easier to retype what you see rather than to create it.

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#166

Post by gmerc »

amelius wrote: post #117
capableweb wrote: post #5

I'm glad that they did, although they should obviously done an announcement for it. The amount of people in the ecosystem who thinks it's even possible to detect if something is AI written or not when it's just a couple of sentences is staggering high. And somehow, people in power seems to put their faith in some of these tools that guarantee a certain amount of truthfulness when in reality it's impossible they could…

They could certainly keep a database of things generated by /their/ AI ...

Which would be trivially broken with emojis injection or viewpoint shifting.

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#167

Post by mplewis »

pixl97 wrote: post #85
hackernewds wrote: post #15

Good. If it is not reliable, it's a further harm than good if it exists as a false security. An analogous example: my local pizza delivery (where I worked) would shut the box with a safety sticker, to avoid tampering / dipping by the delivery boys. Now, sometimes they would forget to do this for various logistical reasons. Every one of the non-stickered ones started getting returned as customers worried a pepperoni s…

Eh, I'd consider that a failure of employee training and reverse the situation by giving out a weekly bonus to shifts that did not fail to put the security stickers on. Kinda like if they forgot to put the security seal on your aspirin, I'm not going to take them all off because someone forgot to run production with all the bottles sealed.

Frankly, the kind of person who forgets to put the sticker on at the pizza place will forget about the bonus too.

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#168

Post by mplewis »

ribosometronome wrote: post #153
jedberg wrote: post #151

Latest I heard is that teachers are requiring homework to be turned in in Google Docs so that they can look at the revision history and see if you wrote the whole thing or just dumped a fully formed essay into GDocs and then edited it. Of course the smart student will easily figure out a way to stream the GPT output into Google Docs, perhaps jumping around to make "edits". A clever and unethical student is pretty muc…

Retyping the essay from the chatgpt while actively rewording the occassional sentence seems like it would do it.

It's a bit suspicious to type an essay linearly from start to finish, though.

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#169

Post by RandomLensman »

constantcrying wrote: post #64
capableweb wrote: post #5

I'm glad that they did, although they should obviously done an announcement for it. The amount of people in the ecosystem who thinks it's even possible to detect if something is AI written or not when it's just a couple of sentences is staggering high. And somehow, people in power seems to put their faith in some of these tools that guarantee a certain amount of truthfulness when in reality it's impossible they could…

Even the idea of it is bad, ChatGPT is supposed to write indistinguishably from a human. The "detector" has extremely little information and the only somewhat reasonable criteria are things like style, where ChatGPT certainly has a particular, but by no means unique writing style. And as it gets better it will (by definition) be better at writing in more varied styles.

Why even care if it is written by a machine or not? I am not sure it matters as much as people think.

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#170

Post by constantcrying »

bko wrote: post #157
constantcrying wrote: post #64

Earlier quoted context omitted.

Even the idea of it is bad, ChatGPT is supposed to write indistinguishably from a human. The "detector" has extremely little information and the only somewhat reasonable criteria are things like style, where ChatGPT certainly has a particular, but by no means unique writing style. And as it gets better it will (by definition) be better at writing in more varied styles.

Copying a comment I posted a while ago: I listened to a podcast with Scott Aaronson that I'd highly recommend [0]. He's a theoretical computer scientist but he was recruited by OpenAI to work on AI safety. He has a very practical view on the matter and is focusing his efforts on leveraging the probabilistic nature of LLMs to provide a digital undetectable watermark. So it nudges certain words to be paired together sl…

I think the chance of this working reliably is precisely zero. There are multiple trivial attacks against this and it can not work if the user has any kind of access to token level data (where he could trivially write his own truly random choice). And if there is a non-water marking neural network with enough capacity to do simple rewriting you can easily remove any watermark or the user does the minor rewrite himself.
Post reply on HN