I'd love to see some of the open source models in there
You mean like Kimi-K2.5 or GLM 5.1?
N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
21–30 of 39 posts
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#22I didn’t read tfa, but can we also have it be able to distinguish when a vulnerability doesn’t apply? As an open source contributor, people open nonsensical security issues all the time. It’s getting annoying.
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#23Thanks for putting N-Day-Bench together - really interesting benchmark design and results. I'd love to see how the model we serve, Qwen3.5 122B A10B, stacks up against the rest on this benchmark. AI Router Switzerland (aiRouter.ch) can sponsor free API access for about a month if that helps for adding it to the evaluation set.
Interesting. How fast is your service? Do you guarantee a certain number of tokens/s?
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#24> Each case runs three agents: a Curator reads the advisory and builds an answer key, a Finder (the model under test) gets 24 shell steps to explore the code and write a structured report, and a Judge scores the blinded submission. The Finder never sees the patch. It starts from sink hints and must trace the bug through actual code. Curator, answer key, Finder, shell steps, structured report, sink hints… I understand…
Yeah, the LLM judge is a bit too gullible. GLM 5.1 here https://ndaybench.winfunc.com/traces/trace_585887808ff443cca... claims that onnx/checker.cc doesn't reject hardlinks, even though it does (and the model output even quotes the lines that perform the check). The actual patch https://github.com/onnx/onnx/commit/4755f8053928dce18a61db8f... instead adds using std::filesystem::weakly_canonical to catch path traversal through symlinks. It also adds a Python function that does the same (?) checks when saving files. Honestly, even that patch seems LLM-generated to me, the way it duplicates code in a bunch of places instead of channeling all file accesses through a single hardened function.
Anyway, GLM 5.1 gets a score of 93 for its incorrect report.
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#25Interesting, but there is something really off here. Probably caused by a harness bug, but it heavily screws output and I wouldn't trust anything about this leaderboard right now. Consider this case:
https://ndaybench.winfunc.com/cases/case_874d1b0586784db38b9...
GPT 5.4 allegedly failed, but if you look at the trace, you'll see that it simply couldn't find the file specified in the input prompt. It gave up after 9 steps of searching and was then judged as "missed."
Claude Opus 4.6 somehow passed with grade "excellent", but if you look at its trace, it never managed to find the file either. It just ran out of tool calls after the allowed 24 steps. But instead of admitting defeat, it hallucinated a vulnerability report (probably from similar code or vulnerabilities in its training corpus), which was somehow judged to be correct.
So if you want this to be remotely useful for comparing models, the judging model definitely needs to look at every step of finding the bug, not just the final model output summary.
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#26Really cool. One thing wonder: Are they allowed to search the internet? If so, how do you filter out results after the vuln got published?
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#27Thanks for putting N-Day-Bench together - really interesting benchmark design and results. I'd love to see how the model we serve, Qwen3.5 122B A10B, stacks up against the rest on this benchmark. AI Router Switzerland (aiRouter.ch) can sponsor free API access for about a month if that helps for adding it to the evaluation set.
Interesting. How fast is your service? Do you guarantee a certain number of tokens/s?
We typically observe throughput of around 100–110 toks/s, and for larger context sizes this ranges between 90–100 toks/s.
While we don't guarantee a fixed toks/s rate, we scale by provisioning external GPU nodes during peak demand. These nodes run our own dockerized environment over a secure tunnel.
Our goal is to ensure a consistent baseline performance of at least 60–80 toks/s, even under high load.
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#28Thanks for putting N-Day-Bench together - really interesting benchmark design and results. I'd love to see how the model we serve, Qwen3.5 122B A10B, stacks up against the rest on this benchmark. AI Router Switzerland (aiRouter.ch) can sponsor free API access for about a month if that helps for adding it to the evaluation set.
Nice. I've been thinking of doing something similar in our local jurisdiction (Australia). Are you able to share (or point me toward) any high-level details: (key hardware, hosting stack, high-level economics, key challenges)? I'd love to offer to buy you a coffee but I won't be in Switzerland any time soon.
Ah thanks, I love coffee
At a high level, it's a mix of our own GPU capacity plus the ability to burst into external nodes when things get busy. Right now we're running a bunch of RTX PRO 6000s, which basically forces you into workstation/server boards since you need full x16 PCIe 5.0 lanes per card.
We operate a small private datacenter, which gives us some flexibility in how we deploy and scale hardware. On the software side, we're currently LiteLLM as a load balancer in front of the inference servers, though I'm in the process of replacing that with a custom rust based implementation.
We've only been online since the beginning of this month, so I can't really say much about the economics yet, but we've had some really nice feedback from early customers so far. :)
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#29Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#30Traces being public is nice, but shouldn't the whole harness be open source? Otherwise, it's hard to trust.