Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: AI Code Detector – detect AI-generated code with 95% accuracy

code-detector.ai

Post reply on HN
Print view View original

11–20 of 69 posts

Re: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy

#12

Post by johnsillings »

Ndotkess wrote: post #7

What is your approach to measuring accuracy?

I'm sure Henry will chime in here, but there's some more info here in the technical announcement: https://www.span.app/introducing-span-detect-1

"span-detect-1 was evaluated by an independent team within Span. The team’s objective was to create an eval that’s free from training data contamination and reflecting realistic human and AI authored code patterns. The focus was on 3 sources: real world human, AI code authored by Devin crawled from public GitHub repositories, and AI samples that we synthesized for “brownfield” edits by leading LLMs. In the end, evaluation was performed with ~45K balanced datasets for TypeScript and Python each, and an 11K sample set for TSX."

Re: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy

#13

Post by henryl »

jensneuse wrote: post #9

Could I use this to iterate over my AI generated code until it's not detectable anymore? So essentially the moment you publish this tool it stops working?

I'm sure you can but there isn't really an adversarial motive for doing that, I would think :)

Re: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy

#14

Post by henryl »

JohnFriel wrote: post #6

This is interesting. Do you know what features the classifier is matching on? Like how much does stuff like whitespace matter here vs. deeper code structure? Put differently, if you were to parse the AI and non-AI code into AST and train a classifier based on that, would the results be the same?

Candidly, it's a bit of a black box still. We hope to do some ablation studies soon, but we tried to have a variety of formatting and commenting styles represented in both training and evaluation.

Re: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy

#15

Post by henryl »

jfarina wrote: post #5
henryl wrote: post #3

Earlier quoted context omitted.

I'd argue that knowing AI generated code shipped into production is the first step to understanding the impact of AI coding assistants on velocity and quality. When paired with additional context, it can help leaders understand how to improve proficiency around these tools.

That's not relevant to the comment you replied to.

Ah - I misread:

Recall 91.5, F1 93.3

Re: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy

#16

Post by well_actulily »

jensneuse wrote: post #9

Could I use this to iterate over my AI generated code until it's not detectable anymore? So essentially the moment you publish this tool it stops working?

This is essentially the adversarial generator/discriminator set-up that GANs use.

Re: Show HN: AI Code Detector – detect AI-generated code with 95% accuracy

#19

Post by mannicken »

Only Python, TypeScript and JavaScript? Well there go my vibe-coded elisp scripts.

I guess it's impossible (or really hard) to train a language-agnostic classifier.

Reference, from your own URL here: https://www.span.app/introducing-span-detect-1

Post reply on HN