31–40 of 44 posts
I had to read this sentence twice: Right now many niche book communities are facing an onslaught of unlabeled AI works, mine included. On first reading, I thought the author meant that they themselves had submitted 'unlabeled AI works' to one or more niche book communities.
[0] https://archive.org/search?query=analog%20science%20fiction%...
Note that any good binary classifier (like "this is AI writing" for Pangram) should maximize the following four probabilities: 1. P(actually true | predicted true) (= true positive rate, recall, sensitivity) 2. P(actually false | predicted false) (= true negative rate, specificity) 3. P(predicted true | actually true) (= positive predictive value, precision) 4. P(predicted false | actually false) (= negative predicti…
They report false negative rate (FNR) here: https://www.pangram.com/research/model-card/pangram-3-3 Academic Writing FNR=0.00% N=48443 Creative Writing FNR=0.23% N=41940 RAID FNR=0.93% N=66855 Recall = 1 - FNR
I had to read this sentence twice: Right now many niche book communities are facing an onslaught of unlabeled AI works, mine included. On first reading, I thought the author meant that they themselves had submitted 'unlabeled AI works' to one or more niche book communities.
How did reading the sentence twice help? Just from that sentence, there isn’t anything that disambiguates it.
The second read was deliberate, and motivated by trying to understand my initial confusion. I considered the sentence structure and realized that 'mine included' could, if you ignore strict grammar rules, refer to 'communities' (which made more sense in context) and not 'works'.
I’m a teacher that requires students to write, so I’m always on the lookout for reliable AI detection. Haven’t anything close to reliable yet, but Pangram definitely seemed to be better than anything else in my (very limited) tests. That said, it did incorrectly flag something that I personally wrote as AI, which baffled me. I spent a while playing around with the text and feeding it back into Pangram, trying to figu…
> Pangram boasts a false positive rate of 1 in a 10,000. That is, if Pangram says a block of text is AI there is only a one in ten thousand chance that it was written by a human. That'd be if they had a false discovery rate of 1/10,000. If for instance: * 100,000 samples are tested * 100 of which are AI-generated, the rest human-written * Pangram flags 50 of the AI-generated samples (true positives) * Pangram also fl…
In your scenario, False negatives = 50 False negative rate = FN / (TP + FN) = 50 / (50 + 50) = 50% Their actual FNR is much better than 50%, more like 1% at worst.
But as the parent says, this means you're still making plenty of mistakes. You're just not making the specific kind of mistake you're trying hard not to make, though I'm pretty skeptical of anyone claiming 1% false positives. It's hard to even envision a truly valid way of assessing this. You'd at minimum need an adversarial test set created by an independent third party that has no incentive to be favorable to you, and even then, the landscape is so constantly shifting that an error rate estimated at one time tells you not exactly nothing about a future time, but this isn't exactly estimating the strength of gravity. You can be correct one week and wrong the next.
Not a statistical estimate, but getting shot in the head doesn't even have a 99% kill rate.
I had to read this sentence twice: Right now many niche book communities are facing an onslaught of unlabeled AI works, mine included. On first reading, I thought the author meant that they themselves had submitted 'unlabeled AI works' to one or more niche book communities.
How did reading the sentence twice help? Just from that sentence, there isn’t anything that disambiguates it.
Pangram is witchcraft to me. The way they can correctly detect AI writing from small samples, with so little statistical signal, is crazy. I've seen it correctly detect AI writing even when people use those "humanizer" rewriting skills that remove the hallmarks of AI writing and make the text indistinguishable from human text. But not to Pangram.
Those skills are meant for humans, not for models. Pangram said in their recent model update that they now also detect the humanizers.
All it requires on their end is running each of their AI outputs through each humanizer and including it in their corpus as ai-humanized.
It's essentially impossible now to prompt your way to non AI detectable text. Even if you do today, it'll be picked up by the next model update.
This is why in academic environments its probably worth re-testing old exams or papers periodically - same way blood and urine samples from athletes are preserved to take advantage of better future testing.