Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI shuts down its AI Classifier due to poor accuracy

decrypt.co

Post reply on HN
Print view View original

111–120 of 292 posts

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#111

Post by pixl97 »

frumper wrote: post #91
pixl97 wrote: post #85

Earlier quoted context omitted.

Eh, I'd consider that a failure of employee training and reverse the situation by giving out a weekly bonus to shifts that did not fail to put the security stickers on. Kinda like if they forgot to put the security seal on your aspirin, I'm not going to take them all off because someone forgot to run production with all the bottles sealed.

The bottle of Aspirin goes through many hands between the manufacturer and you including sitting on an unattended shelf open to the public. The person making the pizza is working for the same company as the person delivering it, or may even be the same person. If you can't trust the pizza co delivery person then you probably shouldn't trust the person making it either.

Your right, don't eat at that pizza place either

This is the brown m&m principle in effect.

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#112

Post by JustBreath »

siglesias wrote: post #110
constantcrying wrote: post #64

Earlier quoted context omitted.

Even the idea of it is bad, ChatGPT is supposed to write indistinguishably from a human. The "detector" has extremely little information and the only somewhat reasonable criteria are things like style, where ChatGPT certainly has a particular, but by no means unique writing style. And as it gets better it will (by definition) be better at writing in more varied styles.

I'd challenge this assumption. ChatGPT is supposed to convey information and answer questions in a manner that is intelligible to humans. It doesn't mean it should write indistinguishably from humans. It has a certain manner of prose that (to me) is distinctive and, for lack of a better descriptor, silkier, more anodyne, than most human writing. It should only attempt a distinct style if prompted to.

That's true, you could even purposely inject fingerprinting into its writing style and it could still accomplish the goal of conveying information to people.

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#113

Post by mananaysiempre »

tyingq wrote: post #77
bgirard wrote: post #49

Earlier quoted context omitted.

> A tool that gives incorrect and inconsistent results shouldn’t have any part of a decision making process. It can be used for some decision (i.e. not critical ones), but it should NOT be used to accused someone of academic misconduct unless the tool meets a very robust quality standard. > this tool is as reliable as a magic 8-ball Citation needed

> "should NOT be used to accused someone of academic misconduct unless the tool meets a very robust quality standard." Meanwhile, the leading commercial tools for plagiarism detection often flag properly cited/annotated quotes from sources in your text as plagiarism.

That sounds like a less serious problem—if the tool highlights the allegedly plagarized sections, at worst the author can conclusively prove it false with no additional research (though that burden should instead be on the tool’s user, of course). So it’s at least possible to use the tool to get meaningful results.

On the other hand, an opaque LLM detector that just prints “that was from an LLM, methinks” (and not e.g. a prompt and a seed that makes ChatGPT print its input) essentially cannot be proven false by an author who hasn’t taken special precautions against being falsely accused, so the bar for sanctioning people based on its output must be much higher (infinitely so as far as I am concerned).

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#114

Post by csmpltn »

pixl97 wrote: post #109
csmpltn wrote: post #47

Earlier quoted context omitted.

Why is this comment being downvoted? OpenAI can internally keep a "hash" or a "signature" of every output it ever generated. Given a piece of text, they should then be able to trace back to either a specific session (or a set of sessions) through which this text was generated in. Depending on the hit rate and the hashing methods used, they may be able to indicate the likelihood of a piece of text being generated by A…

Why would they want to is my question. A single character change would break it. Then you have database costs of storing all that data forever. Moreso, it's only for openAI, I don't think it will be too long before other gpt4 level models are around and won't give two shits about catering to the AI identification police.

> A single character change would break it.

That depends on how they hash the data, right? They can use various types of Perceptual Hashing [1] techniques which wouldn't be susceptible to a single-character change.

[1] https://en.wikipedia.org/wiki/Perceptual_hashing

> Then you have database costs of storing all that data forever.

A database of all textual content generated by people? That sounds like a gold mine, not a liability. But as I've mentioned earlier, they don't need to keep the raw data (a perceptual hash is enough).

> won't give two shits about catering to the AI identification police

I'm sure there will be customers willing to pay for access to these checks, even if they're only limited to OpenAI's product (universities and schools - for plagiarism detection, government agencies, intelligence agencies, police, etc).

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#115

Post by Ukv »

hayd wrote: post #60

Was this what Stack Overflow were using to detect automated answers?

None were officially built into the site so it'll vary from moderator to moderator, but the one that mods had a browser script made for to help streamline moderation was RoBERTa Base OpenAI Detector from 2019, created prior to the existence of GPT-3, GPT-3.5 (ChatGPT free), or GPT-4 (ChatGPT pro). It'll be far worse than the 2023 one this article is about.

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#116

Post by jerf »

squarefoot wrote: post #87

How could we have both the AI that is indistinguishable from humans and the AI that can detect it with good accuracy? That would imply the race on both sides for an AI that is more intelligent than an high IQ human.

Text is a very high dimensional space, n-dimensional in fact. There is plenty of room for an AI to leave a fingerprint that can be detected in some ways but not others.

In fact it doesn't take much text to distinguish between two human beings. The humanly-obvious version is that someone that habitually speaks in one dialect and someone else in another must be separate, but even without such obvious tells humans separate themselves into characterizeable subsets of this space fairly quickly.

I'm skeptical about generalized AI versus human detection in the face of the fact that it is adversarial. But a constant, unmoving target of some specific AI in some particular mode would definitely be detectable; e.g., "ChatGPT's current default voice" would certainly be detectable, "ChatGPT when instructed to sound like Ernest Hemmingway" would be detectable, etc. I just question whether ChatGPT in general can be characterized.

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#117

Post by amelius »

capableweb wrote: post #5

I'm glad that they did, although they should obviously done an announcement for it. The amount of people in the ecosystem who thinks it's even possible to detect if something is AI written or not when it's just a couple of sentences is staggering high. And somehow, people in power seems to put their faith in some of these tools that guarantee a certain amount of truthfulness when in reality it's impossible they could…

They could certainly keep a database of things generated by /their/ AI ...

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#118

Post by cwkoss »

rootusrootus wrote: post #81

Good. And I think watermarking AI output is also a dead end. Better that we simply assume that all content is fake unless proven otherwise. To the extent that we need trustworthy photos, it seems like a better idea to cryptographically sign images at the hardware level when the photo is taken. Voluntarily watermarking AI content is completely pointless.

Cryptography cant save us here, people will figure out how to send AI images to the crypto hardware to get it signed in months. Just would be another similar layer of false security.

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#119

Post by littlestymaar »

JustBreath wrote: post #112
siglesias wrote: post #110

Earlier quoted context omitted.

I'd challenge this assumption. ChatGPT is supposed to convey information and answer questions in a manner that is intelligible to humans. It doesn't mean it should write indistinguishably from humans. It has a certain manner of prose that (to me) is distinctive and, for lack of a better descriptor, silkier, more anodyne, than most human writing. It should only attempt a distinct style if prompted to.

That's true, you could even purposely inject fingerprinting into its writing style and it could still accomplish the goal of conveying information to people.

But then that wouldn't be “detecting AI”, but merely recognizing an intentionally added fingerprint, which sounds far less attractive…

Re: OpenAI shuts down its AI Classifier due to poor accuracy

#120

Post by SkyPuncher »

whimsicalism wrote: post #83
xattt wrote: post #9

Earlier quoted context omitted.

I see no reason why watermarking can’t be broken by having someone simply rephrase/redraw the output. Yes, it’s still work, but it’s one step removed from having to think up of the original content.

Watermarking was never going to be successful except for the most naive uses.

It can likely work in images where you can make subtle, human-undetectable tweaks across thousands/millions of pixels, each with many possible values.

Nearly impossible across data with a couple hundred characters and dozens to thousands of tokens.

Post reply on HN