Consider trying out GPTKit https://gptkit.ai it has higher accuracy than GPTZero.
Detect ChatGPT Generated Content
91–100 of 109 posts
Re: Detect ChatGPT Generated Content
#92Consider trying out GPTKit https://gptkit.ai it has higher accuracy than GPTZero.
Disclosure: The tool is built by me. It uses 6 distinct methods to classify text with a 93% accuracy rate, based on testing a dataset of 100k+ samples.
Re: Detect ChatGPT Generated Content
#93Earlier quoted context omitted.
> The problem with such things is that you would need 100% accuracy in most cases for it to be useful. Why? If there is 75% confidence a students report was generated using ChatGPT, then that’s enough to sit down with the student and discuss the content in person and see if they actually know it. A tool such as this could help the teacher having to avoid doing this with every student, and also reinforce to students t…
This is such a short-sighted view. Students who wish to cheat using ChatGPT will immediately start running their text through these tools preemptively, changing some phrases here and there and add spelling errors to ensure they don't get caught. You're left with innocent students being accused of cheating on regular basis. Maybe the education system will finally learn that asking students to merely recite information…
> changing some phrases here and there and add spelling errors to ensure they don't get caught.
The students who will do this already do this though, just with different source material I suppose.
> Maybe the education system will finally learn that asking students to merely recite information that can be found by anyone, anywhere
I think we’ll see an increased amount of face to face assignments like exams or essays written in class only, where students can’t use these tools. It solves the problem entirely and is arguably better - but increases education cost.
Re: Detect ChatGPT Generated Content
#94Earlier quoted context omitted.
Nothing, this is how adversarial training works, but it also works both ways.
It works both ways, but generation is advantaged in the long run. There has to actually be a statistical difference to detect, and AI outputs without statistical differences from human output are obviously possible, since humans make them all the time.
Sort off, I’m aware that in principle the generator has an advantage and eventually the detector will average out to a coin flip at best.
However some advantages can disappear when you put constraints on the output such as quality and correctness.
So whilst the end result might be less statistically significant in terms of was it human or AI generated it can overall be also less useful to the end user.
Re: Detect ChatGPT Generated Content
#95Earlier quoted context omitted.
That’s a very naïve outlook in reality like with every system the burden of proof would end up falling on the accused not the accuser…
The reality in my experience (as a university lecturer) is that the burden of proof very much falls on the accuser rather than the accused.
How is that? Students already are required to submit a lot of evidence with their work such as research notes, plans, lab work etc. to prove they’ve actually done that and if plagiarism detection systems flag any of their work they have to defend it rather than the institution having to investigate and build evidence on its own outside of w/e shoddy plagiarism detection system they bought told them.
Re: Detect ChatGPT Generated Content
#96Consider trying out GPTKit https://gptkit.ai it has higher accuracy than GPTZero.
Disclosure: The tool is built by me. It uses 6 distinct methods to classify text with a 93% accuracy rate, based on testing a dataset of 100k+ samples.
Where are you hosting the dataset? Would love to help out, I'm building an open source data version control tool to help iterate on ML datasets.
https://github.com/Oxen-AI/oxen-release
Would be cool if we could get a community around the test dataset to insure that 93% accuracy rate. Then people can add their failure cases to the repo and then you can iterate on them.
Re: Detect ChatGPT Generated Content
#97We ought to require OpenAI to run something like this: a “Hey ChatGPT, this you?” endpoint that replies Yes/No/similarity score, and a creation timestamp, when you hit it with some text. As regulations go, this one’s not too burdensome: expensive, but pretty cheap compared to training and running a large language model in the first place.
This works right up until the model is open sourced or otherwise replicable.
Put another way, it only works in today's environment.
When ML hardware is as widely distributed as classical compute, and all the models are on HuggingFace, you will be back to 0
Re: Detect ChatGPT Generated Content
#98The problem with such things is that you would need 100% accuracy in most cases for it to be useful. For example many schools and universities fear that students use ChatGPT for homework. If such a plagiarism checker has false positive results in just a small percentage of cases, the consequences for honest students would be too severe to actually act on the results of the check. But perfect accuracy can never be rea…
At the university level, the problem of cheating is more a problem with our idea of a university. In an ideal world, you go to a university to learn. If someone wants to cheat, ultimately they're just cheating themselves (and, if at a private university or even most public universities, throwing a lot of money away). "Cheating" isn't really the university's problem in that view. Ofc, in reality universities aren't (o…
[deleted]
Re: Detect ChatGPT Generated Content
#99We ought to require OpenAI to run something like this: a “Hey ChatGPT, this you?” endpoint that replies Yes/No/similarity score, and a creation timestamp, when you hit it with some text. As regulations go, this one’s not too burdensome: expensive, but pretty cheap compared to training and running a large language model in the first place.
This works right up until the model is open sourced or otherwise replicable. Put another way, it only works in today's environment. When ML hardware is as widely distributed as classical compute, and all the models are on HuggingFace, you will be back to 0
If it buys some time, that’s a good day’s work and I’ll be very satisfied. All you can really ask is that a regulation makes things better in the near term.
The future will know a lot more about the nature and implications of persuasively-humanlike ML than I do: it can take care of itself. Maybe by then hybrid writing will be the norm and not considered plagiarism, and we’ll all have trustworthy virtual assistants shielding us from scams. But in the meantime, there are some reasonable causes for concern, and this would help.
Re: Detect ChatGPT Generated Content
#100This only calculates perplexity and burstiness. I don't think that's going to work very well. It would be much better to try and detect whether the distribution from which a piece of text was drawn is closer to that of a human, or that of a large language model.
But how would one go about detecting something like that? Well, one would need a model of human language trained to approximate the distribution of tokens in a large corpus of natural language... text...
Oh wait.