Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

ndaybench.winfunc.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

#31

Post by croemer »

Heavily vibe coded, the judge can even change the weights and that's presented as a feature ("conscious tradeoff"), see methodology section 7:

> The rubric is fixed across all cases. Five dimensions, weighted: target alignment (30%), source-to-sink reasoning (30%), impact and exploitability (20%), evidence quality (10%), and overclaim control (10%).

> There's no server-side arithmetic that recomputes the overall score from dimension scores and weights. The Judge LLM produces the entire score object in one pass. This is a conscious trade-off: it avoids the brittleness of post-hoc formula application at the cost of giving the Judge more interpretive latitude than a mechanical scorer would have.

How on earth is a post-hoc formula application "brittle"? Classic LLM giving bogus reasons instead of the real ones (laziness).

Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

#34

Post by Aurornis »

sigmoid10 wrote: post #25

Interesting, but there is something really off here. Probably caused by a harness bug, but it heavily screws output and I wouldn't trust anything about this leaderboard right now. Consider this case: https://ndaybench.winfunc.com/cases/case_874d1b0586784db38b9... GPT 5.4 allegedly failed, but if you look at the trace, you'll see that it simply couldn't find the file specified in the input prompt. It gave up after 9 s…

Good find. This appears to be another vibe coded vanity project where the output was never checked.

All of the online spaces where LLMs are discussed are having a problem with the volume of poorly vibecoded submissions like this. Historically I’ve really enjoyed Show HN type submissions but this year most of the small projects that get shared here and on other social medias turn out to be a waste of my time due to all of the vibecoding and how frequently the projects don’t do what they say they do when you look into the details.

Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

#36

Post by phantomoc »

sigmoid10 wrote: post #25

Interesting, but there is something really off here. Probably caused by a harness bug, but it heavily screws output and I wouldn't trust anything about this leaderboard right now. Consider this case: https://ndaybench.winfunc.com/cases/case_874d1b0586784db38b9... GPT 5.4 allegedly failed, but if you look at the trace, you'll see that it simply couldn't find the file specified in the input prompt. It gave up after 9 s…

[dead]

Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

#37

Post by Tepix »

sacrelege wrote: post #27
Tepix wrote: post #23

Earlier quoted context omitted.

Interesting. How fast is your service? Do you guarantee a certain number of tokens/s?

We typically observe throughput of around 100–110 toks/s, and for larger context sizes this ranges between 90–100 toks/s. While we don't guarantee a fixed toks/s rate, we scale by provisioning external GPU nodes during peak demand. These nodes run our own dockerized environment over a secure tunnel. Our goal is to ensure a consistent baseline performance of at least 60–80 toks/s, even under high load.

Sounds good. I saw that you use the FP8 version of the model. Do you also quantize the KV cache?

Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

#38

Post by sacrelege »

Tepix wrote: post #37
sacrelege wrote: post #27

Earlier quoted context omitted.

We typically observe throughput of around 100–110 toks/s, and for larger context sizes this ranges between 90–100 toks/s. While we don't guarantee a fixed toks/s rate, we scale by provisioning external GPU nodes during peak demand. These nodes run our own dockerized environment over a secure tunnel. Our goal is to ensure a consistent baseline performance of at least 60–80 toks/s, even under high load.

Sounds good. I saw that you use the FP8 version of the model. Do you also quantize the KV cache?

no I don't, since there seem to be a silent degradation bug

Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

#39

Post by rohansood15 »

muldvarp wrote: post #20
rohansood15 wrote: post #12

Earlier quoted context omitted.

I worked in AppSec in the past, made sense to me. Maybe you aren't the target audience? You don't really need manual verification for these, the CVEs (vulnerabilities) are public and can be programmatically validated.

Manual verification that the "judge" judges correctly. Also, how exactly do you programmatically validate CVEs?

Most open-source CVEs will have a patch linked in their disclosure. You can get vulnerable code via the git diff, then just verify if it is part of the LLM's finding.
Post reply on HN