DetectGPT: Zero-Shot Machine-Generated Text Detection
ericmitchell.ai
DetectGPT: Zero-Shot Machine-Generated Text Detection
1–10 of 71 posts
Re: DetectGPT: Zero-Shot Machine-Generated Text Detection
#2Research paper:
https://arxiv.org/abs/2301.11305
Additional explanation:
https://twitter.com/_eric_mitchell_/status/16188203614199152...
Re: DetectGPT: Zero-Shot Machine-Generated Text Detection
#3Research paper: https://arxiv.org/abs/2301.11305 Additional explanation: https://twitter.com/_eric_mitchell_/status/16188203614199152...
Awesome thread by one of the authors of paper, concise and insightful, thanks for sharing !
Research was aiming to bring a tool/approach on distinguishing text from LLM and other sources but in the end of the day it will only benefit those with non-open-source LLMs adjust to such technique and "fool" better everyone else (cause we need log-probs out of a model on each of the sample text).
It seems kinda ironic for me, maybe i missed some crucial point here.
Re: DetectGPT: Zero-Shot Machine-Generated Text Detection
#4I might not be understanding what AUROC really means, but how is this performing in terms of false positives/negatives in comparison to human readers?
Re: DetectGPT: Zero-Shot Machine-Generated Text Detection
#5This approach seems to require knowledge of which LLM was used to generate the given text. I wonder if e.g. model fine-tuning - as already provided by Open AI [0] - could evade this detection approach.
Re: DetectGPT: Zero-Shot Machine-Generated Text Detection
#6How does this compare to GPT2 output detector?
I created AIwritingcheck.org to provide teachers with a user friendly interface for this model.
Re: DetectGPT: Zero-Shot Machine-Generated Text Detection
#7Research paper: https://arxiv.org/abs/2301.11305 Additional explanation: https://twitter.com/_eric_mitchell_/status/16188203614199152...
Awesome thread by one of the authors of paper, concise and insightful, thanks for sharing ! Research was aiming to bring a tool/approach on distinguishing text from LLM and other sources but in the end of the day it will only benefit those with non-open-source LLMs adjust to such technique and "fool" better everyone else (cause we need log-probs out of a model on each of the sample text). It seems kinda ironic for me…
Highly unlikely any popular model would not provide log-probs, since they’re potentially useful. For example:
https://aidungeon.medium.com/controlling-gpt-3-with-logit-bi...
Re: DetectGPT: Zero-Shot Machine-Generated Text Detection
#8How does this compare to GPT2 output detector? I created AIwritingcheck.org to provide teachers with a user friendly interface for this model.
Consider trying out GPTKit https://gptkit.ai it has higher accuracy than GPTZero and uses 6 different methods
Re: DetectGPT: Zero-Shot Machine-Generated Text Detection
#9I might not be understanding what AUROC really means, but how is this performing in terms of false positives/negatives in comparison to human readers?
Here’s a post on understanding AUC (Area Under The Curve) and ROC (Receiver Operating Characteristics):
https://towardsdatascience.com/understanding-auc-roc-curve-6...
Re: DetectGPT: Zero-Shot Machine-Generated Text Detection
#10How does this compare to GPT2 output detector? I created AIwritingcheck.org to provide teachers with a user friendly interface for this model.
Wouldn't GPT-2 be detectable by virtue of being bad? Or is it bad in a way that is more or less indistinguishable from student writing?
It's been a while since I used it but I very rarely got plausible output from it.