[deleted]
AI Detectors Get It Wrong. Writers Are Being Fired Anyway
131–140 of 189 posts
Re: AI Detectors Get It Wrong. Writers Are Being Fired Anyway
#132The one kind of AI detector that could work would be if the AIs store some checksum(s) of each text they write. Then you can ask it if it wrote a certain text. With the naive version of this, you only have to change one word to get around the system. A better version is to checksum smaller segments. Maybe a "chunk size" of 50 words is good. If you find several such chunks in a text, it's pretty clear you have a sligh…
This won't work, as local AI (the one I can run on my budget-friendly laptop without any Internet access) exists today and even beats GPT3.5 in some benchmarks.
It may work partially since most people don't have and don't want to have local LLMs. It's not all or nothing.
Re: AI Detectors Get It Wrong. Writers Are Being Fired Anyway
#133Earlier quoted context omitted.
Yeah but to reach that point you will probably need those "useless AI detectors" (as stated by the comment I was replying to). That was my point - we're not there yet therefore those tools can be useful.
But how do you know we’re not there yet? Not across the board, but isn’t it possible there’s a small yet growing portion of written content online that’s AI generated with no obvious tells?
Semi-automated such is probably widespread by now.
Like imagine the Rust Evangelic Task Force, but for the next big thing, will probably be shilled by bots.
Low quality content like Youtube and Reddit comments are probably mostly LLM bots who comment on anything to hide the actual spam comments.
Re: AI Detectors Get It Wrong. Writers Are Being Fired Anyway
#134we are apparently cursed to forever re-learn the lessons of reverend bayes on detection theory. prostate cancer, airport terrorists, slop detection … you either design your system to handle the off-diagonal parts of the confusion matrix properly or you suffer the consequences.
Unfortunately the people suffering the consequences are not the people who had any input into the design of the system.
Re: AI Detectors Get It Wrong. Writers Are Being Fired Anyway
#135Earlier quoted context omitted.
Based on the article provided, several elements suggest that the narrative could have been written or heavily influenced by an AI. Below are key points from the article that support this suspicion, each backed by direct citations: 1. *Generic Language and Lack of Specific Detail*: The article describes Kimberly Gasuras’s experience with broad, generalized statements that lack specific, nuanced detail that a human wri…
Did you use the prompt I gave exactly? Edit: Well, I tried my prompt with Gemini and now I have a report about a Guardian journalist who is more than likely using AI to write their articles!
The prompt was:
I strongly suspect the story below was written by or with heavy use of an AI. write a report with citations from the article to argue my case.
and then copied the first half of the article
Re: AI Detectors Get It Wrong. Writers Are Being Fired Anyway
#136In some cases an AI will make a weird word choice. So do a lot of humans. Sometimes AIs are needlessly wordy. Um...so are a lot of humans. Rinse and repeat. AI detectors are useless. The AIs are training on human writing, so they write fundamentally like humans. How is this not obvious?
[deleted]
Re: AI Detectors Get It Wrong. Writers Are Being Fired Anyway
#137In some cases an AI will make a weird word choice. So do a lot of humans. Sometimes AIs are needlessly wordy. Um...so are a lot of humans. Rinse and repeat. AI detectors are useless. The AIs are training on human writing, so they write fundamentally like humans. How is this not obvious?
A fairly simple and useful AI detector that works uncannily well on student papers: (a) does the text contain "I am an AI" or words to that effect, (b) are there lots of completely made up references?
[flagged]
Re: AI Detectors Get It Wrong. Writers Are Being Fired Anyway
#138Earlier quoted context omitted.
In the general internet the reputation of AI writing is that it's writing that's bad/awkward in a way that is often identifiable (by humans) as not having been written by humans. AI detectors are useless, you're right, but for the same reason AI is unreliable in other contexts, not because AI writing is reliably passable.
> in a way that is often identifiable (by humans) as not having been written by humans. You should check out reddit sometime. It's been nearly twenty years (not hyperbole) of everyone accusing everyone else of being a bot/shill. Humans are utterly incapable of detecting such things. They're not even capable of detecting Nigerian prince emails as scams. > not because AI writing is reliably passable. "Newspaper editor"…
I think we have plenty of evidence that humans have the ability to understand, while chatbots lack such an ability. Therefore, I'm inclined to think that we don't employ some sort of organic LLM but something completely different.
Re: AI Detectors Get It Wrong. Writers Are Being Fired Anyway
#139Earlier quoted context omitted.
There's still some typicality defined by the prompt though. If you ask for a proof in the style of Shakespeare, you're going to get some "average" Shakespeare. It's kind of embedded in the task definition; you're shifting the reference distribution. If a LLM returned something really unusual for Shakespeare when you didn't ask for it, you'd say it's not performing well. Maybe that's tautological but I think it's what…
> you're going to get some "average" Shakespeare The point was that no, you wont (necessarily) get some "average" shakespeare. A sampler may introduce bias and look for the "above average" shakespeare in the distribution.
[deleted]
Re: AI Detectors Get It Wrong. Writers Are Being Fired Anyway
#140Earlier quoted context omitted.
Did you use the prompt I gave exactly? Edit: Well, I tried my prompt with Gemini and now I have a report about a Guardian journalist who is more than likely using AI to write their articles!
The prompt was: I strongly suspect the story below was written by or with heavy use of an AI. write a report with citations from the article to argue my case. and then copied the first half of the article
You should also try the opposite, have the AI show reasons why it was not written by an AI.