Firstly I think this is neat, but the dam has burst.
This might be great for educational institutions but the idea of people needing to know what everyline does as output feels mute to me in the face of agentic AI.
Show HN: AI Code Detector – detect AI-generated code with 95% accuracy
41–50 of 69 posts
Re: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy
#42I feel like code fed into this detector can be manipulated to increase false positives. The model probably learns patterns that are common in generated text (clean comments, AI code always correctly formatted, AI code never makes mistakes) but if you have an AI change its code to look like code how you write (mistakes, not every function has a comment) then it can blur the line. I think this will be a great tool to g…
This is a spot on observation, the most challenging so far to detect appears to be code produced via tooling usage that is slightly ahead of the overall curve in adoption and practices.
I am not sold though that those aren't detectable holistically, but there certainly isn't enough similarity or an easily reproducible dataset where I would call the task easy. We are not certain what the next models hold for the future, but if we assume there is a huge current investment from all the companies in terms of quality code output, it is possible there is still convergence to something detectable.
Re: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy
#43I feel like code fed into this detector can be manipulated to increase false positives. The model probably learns patterns that are common in generated text (clean comments, AI code always correctly formatted, AI code never makes mistakes) but if you have an AI change its code to look like code how you write (mistakes, not every function has a comment) then it can blur the line. I think this will be a great tool to g…
I tested this idea, using ChatGPT5, I asked this prompt: `create two 1000 line python scripts, one that is how you normally do it, and how a messy undergraduete student would write it.` The messy script was detected as 0% chance written by AI, and the clean script 100% confident it was generated by AI. I had to shorten it for brevity. Happy to share the full script. Here is the chatgpt convo: https://chatgpt.com/shar…
That's a great question + something we've discussed internally a bit. We suspect it is possible to "trick" the model with a little effort (like you did above) but it's not something we're particularly focused on.
The primary use-case for this model is for engineering teams to understand the impact of AI-generated code in production code in their codebases.
Re: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy
#44Would be amazing to have a CLI tool that detects AI generated code (even add it as part of CI/CD pipelines). I'm tired of all the AI trash PRs
It is possible to use this via the command line today. I'll ask Henry to have a look & comment here (or grab a demo and leave a note, we'll give you some more details).
Re: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy
#45Very cool piece of tech, I would suggest putting C on the priority list and then Java. Mainly because Unis and Colleges use one of them or both, so that would be a good use case
Totally – we have support for Java, C#, and Ruby in the works. Edit: since you mentioned universities, are you thinking about AI detection for student work, e.g. like a plagiarism checker? Just curious.
Glad to hear Ruby is being in the list as well!
When it comes to the unis, I was thinking of both AI detection for student work. I mean like plagiarism checkers are common nowadays and the systems I know of just forces every student to upload their work and it compares similarities, one even broke it down to AST level (I think?) for detection so it didn't matter if the students renamed the variables.
But for ai detection, it's still a new area. From what I know, unis just make the students check a field when uploading their work as a contract that they never used ai tools and all is their own work, and after that is up to the teacher to go through their code and see if it looks odd or something. Some even have the students just present their code and make them explain what they did. But as of a tool for ai detection is pretty new, as far as I know.
Re: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy
#46Firstly I think this is neat, but the dam has burst. This might be great for educational institutions but the idea of people needing to know what everyline does as output feels mute to me in the face of agentic AI.
Sadly, this doesn't work on the line-level yet. I know that wasn't the main purpose of your comment, but figured I'd mention that first.
Getting more to the heart of your question: the main use-case for this (and the reason Span developed it) is to understand the impact of AI coding assistants in aggregate for their customers. The explosion of AI-generated code is creating some strange issues that engineering teams need to take into account, but visibility is super low right now.
The main idea is that – with some resolution around which code is AI-authored and human-authored – engineering teams can better understand when and how to deploy AI-generated code (and when not to).
Re: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy
#47I will always write code myself but then sometimes have AI generate a first pass at class and method doc strings. What would happen in this scenario with your tool? Would my code be detected as AI generated because of this or does your tool solely operate on code only?
Great question. The model does look at comments, too.
Re: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy
#48I feel like code fed into this detector can be manipulated to increase false positives. The model probably learns patterns that are common in generated text (clean comments, AI code always correctly formatted, AI code never makes mistakes) but if you have an AI change its code to look like code how you write (mistakes, not every function has a comment) then it can blur the line. I think this will be a great tool to g…
I tested this idea, using ChatGPT5, I asked this prompt: `create two 1000 line python scripts, one that is how you normally do it, and how a messy undergraduete student would write it.` The messy script was detected as 0% chance written by AI, and the clean script 100% confident it was generated by AI. I had to shorten it for brevity. Happy to share the full script. Here is the chatgpt convo: https://chatgpt.com/shar…
On HN, indent four spaces for code block, blank line between and text above.
Like
ThisRe: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy
#49Earlier quoted context omitted.
I tested this idea, using ChatGPT5, I asked this prompt: `create two 1000 line python scripts, one that is how you normally do it, and how a messy undergraduete student would write it.` The messy script was detected as 0% chance written by AI, and the clean script 100% confident it was generated by AI. I had to shorten it for brevity. Happy to share the full script. Here is the chatgpt convo: https://chatgpt.com/shar…
That's a great question + something we've discussed internally a bit. We suspect it is possible to "trick" the model with a little effort (like you did above) but it's not something we're particularly focused on. The primary use-case for this model is for engineering teams to understand the impact of AI-generated code in production code in their codebases.
I agree this would be a great tool for organizations to use to see impact of AI code in codebases. Engineers will probably be too lazy to modify the code enough to make it look less AI. You could probably enhance the robustness of your classifier with synthetic data like this.
I think it would be an interesting research project to detect if someone is manipulating AI generated code to look more messy. This paper https://arxiv.org/pdf/2303.11156 Sadasivan et. al. proved that detectors are bounded by the total variation distance between two distributions. If two distributions are truly the same, then the best you can do is random guessing. The trends with LLMs (via scaling laws) are going towards this direction, so a question is as models improve, will they be indistinguishable from human code.
Be fun to collaborate!
Re: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy
#50Earlier quoted context omitted.
I tested this idea, using ChatGPT5, I asked this prompt: `create two 1000 line python scripts, one that is how you normally do it, and how a messy undergraduete student would write it.` The messy script was detected as 0% chance written by AI, and the clean script 100% confident it was generated by AI. I had to shorten it for brevity. Happy to share the full script. Here is the chatgpt convo: https://chatgpt.com/shar…
On HN, indent four spaces for code block, blank line between and text above. Like This
I appreciate the feedback! I just updated to have the 4 space indentation.