If an AI is potentially putting you out of a job, maybe it's time to rethink your job.
The false positive rate of AI detectors and its effect on freelance writers
111–120 of 171 posts
Re: The false positive rate of AI detectors and its effect on freelance writers
#112There's a lot of people with strong opinions whether these detectors can or cannot work, but I implore you to do the experiment I just did. Go to the top 10 you find in a Google search and paste a sufficiently long sample of your prose. Not a random HN comment - at least 200-400 words of normal, coherent text.
It's a game of cat and mouse in the sense that you can build LLMs specifically optimized for evading the current crop of detectors, but in my testing, they work pretty darn well in the general case. While they might not reliably pick on all LLM text, and while there's sometimes a couple of words in human-generated writing that causes them to output a low but non-zero probability of LLM content, they do not rate human-generated text as "99% AI". Especially not across multiple writing samples.
The most likely story here, I suspect, is that the person leaned on LLMs for commissioned writing and is now trying to save face. The secrecy of the models works both ways, right? And frankly - how often do you see people in HN, or people who do commissioned writing, admit in private that they're using ChatGPT? It's cropping up all over the place.
Note that I'm not commenting on the ethics, fairness, or transparency of tools like that. I'm just saying they work far better than you might be suspecting.
Re: The false positive rate of AI detectors and its effect on freelance writers
#113Weird choice to not disclose the writer in question. Would be very interested to see their articles and judge for myself if they were letting AI do a lot of cleanup on what they were writing.
200 articles in 3 years of journalism seems very prolific. Can anyone speak to whats normal for a career journalist?
Re: The false positive rate of AI detectors and its effect on freelance writers
#114Earlier quoted context omitted.
Surely this is trivial to refute? The professor just needs to have his own text that he generated using AI. When your website says it's not AI... then it's curtains I guess?
The student just needs to find a pre-2022 article that is mistakenly marked as AI by the professor's detector. It's harder but probably not that hard.
Ideally, what the student should probably do is get ahold of as much of the professors own work as possible and feed it through the detector. While pre-AI work might be best for proving that the site can't possibly be reliable, post-AI work is probably better for convincing the professor that the consequences of treating the site as reliable are bad for him.
Re: The false positive rate of AI detectors and its effect on freelance writers
#115Pretty sure we are going to see a rise in the use of Oral Exams. The teachers that don't want to use them will continue to churn out students that score high and don't know how to articulate a damn thing. Which may have been the status quo anyway. Edit: I didn't like my original post
I was one of the students who could nail the exams and assignments I turned in, but then I BOMBED oral exams. Because of crippling anxiety. So how do you account for that? Or autism? Or any other sort of neurological disability?
I'm the opposite - so how would you account for that?
Re: The false positive rate of AI detectors and its effect on freelance writers
#116There's a lot of people with strong opinions whether these detectors can or cannot work, but I implore you to do the experiment I just did. Go to the top 10 you find in a Google search and paste a sufficiently long sample of your prose. Not a random HN comment - at least 200-400 words of normal, coherent text. It's a game of cat and mouse in the sense that you can build LLMs specifically optimized for evading the cur…
Yes, I also would like to find an article pre 2020 where AI detector says 99% AI written, because in my small sample I couldn't find any.
Re: The false positive rate of AI detectors and its effect on freelance writers
#117Legal question: if a writer loses a job due to a false accusal of using AI, would they win a damages lawsuit against the AI detector service if they can prove that AI content detection is knowingly imprecise? That's why those types of services (including OpenAI's initial AI detector) often have huge legal disclaimers saying not to take it as 100% accurate.
I'm not sure it would matter here. In my understanding, the victim was a freelance writer / contractor, not an employee. It kinds sux, but I don't think most job protection laws would apply here irrespective of AI involvement. Obviously this depends a lot on country / jurisdiction.
Defamation, not job protection, the lost freelance job is the source of damages, not the basic legal wrong.
Re: The false positive rate of AI detectors and its effect on freelance writers
#118I've got this suspicion a lot of organizations are using these AI detector results to fire people only to replace them with AI generated content.
Re: The false positive rate of AI detectors and its effect on freelance writers
#119Alternative reading: company determines that paid contractor is producing content that they can’t distinguish from a borderline free LLM, and decides to use it instead. Isn’t it likely that AI (not the detector) is the real threat to freelance writers’ livelihoods?
AI detectors neither reliably detect content that was produced by an AI nor content that could readily be produced by one.
Re: The false positive rate of AI detectors and its effect on freelance writers
#120Earlier quoted context omitted.
If you're not joking, I would be fascinated to hear how you think NFTs might be relevant to this.
It's a certificate of authenticity. You can lie about whether you used AI or not, but your reputation would be demolished if you were found out. You could have proof-of-humanity parties, like key-signing parties. (Partying 'cause you're human seems like a good enough reason.) The reputation would be an asset of yours, a post-monetary currency.
Aah OK, so this is more about signing something in a way that proves you produced it and then saying "I swear I didn't use AI for this, and I'm willing to stake my cryptographic reputation on it".