Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

ndaybench.winfunc.com

Post reply on HN
Print view View original

1–10 of 39 posts

N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

#1

Post by mufeedvh »

N-Day-Bench tests whether frontier LLMs can find known security vulnerabilities in real repository code. Each month it pulls fresh cases from GitHub security advisories, checks out the repo at the last commit before the patch, and gives models a sandboxed bash shell to explore the codebase.

Static vulnerability discovery benchmarks become outdated quickly. Cases leak into training data, and scores start measuring memorization. The monthly refresh keeps the test set ahead of contamination — or at least makes the contamination window honest.

Each case runs three agents: a Curator reads the advisory and builds an answer key, a Finder (the model under test) gets 24 shell steps to explore the code and write a structured report, and a Judge scores the blinded submission. The Finder never sees the patch. It starts from sink hints and must trace the bug through actual code.

Only repos with 10k+ stars qualify. A diversity pass prevents any single repo from dominating the set. Ambiguous advisories (merge commits, multi-repo references, unresolvable refs) are dropped.

Currently evaluating GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, GLM-5.1, and Kimi K2.5. All traces are public.

Methodology: https://ndaybench.winfunc.com/methodology

Live Leaderboard: https://ndaybench.winfunc.com/leaderboard

Live Traces: https://ndaybench.winfunc.com/traces

N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
ndaybench.winfunc.com

Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

#6

Post by mufeedvh »

mbbutler wrote: post #4

It would be helpful to add in some cases that do not contain any vulnerabilities to assess false-positive rate as well.

This is a good idea.

Will incorporate false-positive rates into the rubric from the next run onwards.

At winfunc, we spent a lot of research time taming these models to eradicate false-positive rates (it's high!) so this does feel important enough to be documented. Thanks!

Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

#7

Post by cortesoft »

mbbutler wrote: post #4

It would be helpful to add in some cases that do not contain any vulnerabilities to assess false-positive rate as well.

Any code that is certain that it doesn't have any vulnerabilities is going to be pretty trivial to verify.

Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

#8

Post by Cynddl »

> Each case runs three agents: a Curator reads the advisory and builds an answer key, a Finder (the model under test) gets 24 shell steps to explore the code and write a structured report, and a Judge scores the blinded submission. The Finder never sees the patch. It starts from sink hints and must trace the bug through actual code.

Curator, answer key, Finder, shell steps, structured report, sink hints… I understand nothing. Did you use an LLM to generate this HN submission?

It looks like a standard LLM-as-a-judge approach. Do you manually validate or verify some of the results? Done poorly, the results can be very noisy and meaningless.

Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

#9

Post by junofan »

Cynddl wrote: post #8

> Each case runs three agents: a Curator reads the advisory and builds an answer key, a Finder (the model under test) gets 24 shell steps to explore the code and write a structured report, and a Judge scores the blinded submission. The Finder never sees the patch. It starts from sink hints and must trace the bug through actual code. Curator, answer key, Finder, shell steps, structured report, sink hints… I understand…

> Did you use an LLM to generate this HN submission?

Must have.

> The Finder will never see the patch.

I wasn’t worried that this eval would show the answer to the model before evaluating it. Seems requirements leaked into this post.

Post reply on HN