> For student work I would think it’s reasonable to look for patterns across each student’s work. A single Pangram positive across 10 assignments shouldn’t be enough to bring suspicion upon a student, but when 8 out of 10 of someone’s assignments are coming up as AI it’s time to talk.
People mention Pangram has a low false positive rate, but usually such statistics are over a large population. They tell me that if I check many random works I should get a low number of false positives.
But in your example you aren't checking 10 random works from an assortment of authors. You are checking 10 works from one author.
The example from the teacher suggests that something as simple as a section with 3 bullet points can lead to flagging. When you are checking 10 works from one author all it might take for them to get most of them flagged even if they are 100% human written is for that author to have some style choice, like those 3 bullet point sections, that they just use in much of their writing.
This raises the question of what a teacher should do if they talk to the kid and the kid says it was all human written. Should they do what that teacher did when his own written was falsely flagged and start figuring out what tweaks would get it go pass?
That might help if by doing that you find out that all the 8 flagged items have one simple tweak that clears them up. Then I think you have to give the kid the benefit of the doubt and act is if their style just clashed with the detector.
But if different ones take different tweaks to fix that doesn't really say much, because it seems unlikely a detector would have just a single weakness (like the 3 bullet lists) that is good at false positives.
As I mentioned it is usually error rates on random populations that get quoted at least when I see articles about AI detectors. Have any of the leading checkers published research on the distribution of error rates when checkers are used on multiple works from a single individual?