51–60 of 69 posts
I will always write code myself but then sometimes have AI generate a first pass at class and method doc strings. What would happen in this scenario with your tool? Would my code be detected as AI generated because of this or does your tool solely operate on code only?
Great question. The model does look at comments, too.
An AI code detector would be a binary text classifier - you input some text and the output is either "code" or "not-code". This is an "AI AI code detector". You could call it a meta-AI code detector but people might think that's a detector for AI code written by the company formerly known as Facebook.
With "code" or "not-code" did you make a cheeky reference to "hotdog" "not hotdog"?
Earlier quoted context omitted.
I tested this idea, using ChatGPT5, I asked this prompt: `create two 1000 line python scripts, one that is how you normally do it, and how a messy undergraduete student would write it.` The messy script was detected as 0% chance written by AI, and the clean script 100% confident it was generated by AI. I had to shorten it for brevity. Happy to share the full script. Here is the chatgpt convo: https://chatgpt.com/shar…
That's a great question + something we've discussed internally a bit. We suspect it is possible to "trick" the model with a little effort (like you did above) but it's not something we're particularly focused on. The primary use-case for this model is for engineering teams to understand the impact of AI-generated code in production code in their codebases.
Does that provide an incentive for people writing manually to write worse code, structured badly, as proof that they didn't use AI to generate their code?
Is there now a disincentive for writing good code with good comments?
[1] - https://chatgpt.com/share/e/68c9d578-8290-8007-93f4-4b178369...
I wonder how many false positives it has