[dead]
N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
11–20 of 39 posts
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#12> Each case runs three agents: a Curator reads the advisory and builds an answer key, a Finder (the model under test) gets 24 shell steps to explore the code and write a structured report, and a Judge scores the blinded submission. The Finder never sees the patch. It starts from sink hints and must trace the bug through actual code. Curator, answer key, Finder, shell steps, structured report, sink hints… I understand…
I worked in AppSec in the past, made sense to me. Maybe you aren't the target audience?
You don't really need manual verification for these, the CVEs (vulnerabilities) are public and can be programmatically validated.
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#13> Each case runs three agents: a Curator reads the advisory and builds an answer key, a Finder (the model under test) gets 24 shell steps to explore the code and write a structured report, and a Judge scores the blinded submission. The Finder never sees the patch. It starts from sink hints and must trace the bug through actual code. Curator, answer key, Finder, shell steps, structured report, sink hints… I understand…
Is this really that hard to parse?
Curator and Finder are the names of the agents. "answer key" - haven't you ever taken a test in high school? It's an explanation of the answer. "shell steps" I presume means it gets to run 24 commands on the shell. "structured report" - do I really need to explain to you what a report is? "sink hints" - I admit I didn't know this one, but a bit of searching indicates that it's a hint at where the vulnerability lies.
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#14I'd love to see some of the open source models in there
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#15[flagged]
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#16Thanks for putting N-Day-Bench together - really interesting benchmark design and results.
I'd love to see how the model we serve, Qwen3.5 122B A10B, stacks up against the rest on this benchmark. AI Router Switzerland (aiRouter.ch) can sponsor free API access for about a month if that helps for adding it to the evaluation set.
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#17[dead]
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#18Do you plan on adding more models in the future? I would love to see how other OSS modles like Gemma, GPT-OSS and Qwen fare.
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#19Thanks for putting N-Day-Bench together - really interesting benchmark design and results. I'd love to see how the model we serve, Qwen3.5 122B A10B, stacks up against the rest on this benchmark. AI Router Switzerland (aiRouter.ch) can sponsor free API access for about a month if that helps for adding it to the evaluation set.
Nice. I've been thinking of doing something similar in our local jurisdiction (Australia).
Are you able to share (or point me toward) any high-level details: (key hardware, hosting stack, high-level economics, key challenges)?
I'd love to offer to buy you a coffee but I won't be in Switzerland any time soon.
Re: N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
#20> Each case runs three agents: a Curator reads the advisory and builds an answer key, a Finder (the model under test) gets 24 shell steps to explore the code and write a structured report, and a Judge scores the blinded submission. The Finder never sees the patch. It starts from sink hints and must trace the bug through actual code. Curator, answer key, Finder, shell steps, structured report, sink hints… I understand…
I worked in AppSec in the past, made sense to me. Maybe you aren't the target audience? You don't really need manual verification for these, the CVEs (vulnerabilities) are public and can be programmatically validated.
Manual verification that the "judge" judges correctly.
Also, how exactly do you programmatically validate CVEs?