Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vulnerability research is cooked

sockpuppet.org

Post reply on HN
Print view View original

71–80 of 181 posts

Re: Vulnerability research is cooked

#71

Post by MajesticHobo2 »

phyzome wrote: post #50
MajesticHobo2 wrote: post #14

Earlier quoted context omitted.

That was then, this is now. The new models are scarily good. If you're skeptical, just take an hour to replicate the strategy the article references. Point Claude at any open-source codebase you find interesting and instruct it to find exploitable vulnerabilities. Give it a well-defined endpoint if you want (e.g., "You must develop a Python script that triggers memory corruption via a crafted request") and see how we…

> That was then, this is now. No, what we were seeing with curl was script kiddies. It wasn't about the quality of the models at all. They were not filtering their results for validity.

It was definitely partially about model quality. The frontier models are capable of producing valid findings with (reasonably) complex exploit chains on the first pass (or with limited nudging) and are much less prone to making up the kinds of nonsensical reports that were submitted to curl. Compared to now, the old models essentially didn't work for security.

If those script kiddies had been using today's models instead and _still_ didn't do any filtering, a lot more of those bugs would have been true positives.

Re: Vulnerability research is cooked

#72

Post by jgeralnik »

streetfighter64 wrote: post #23

> Is the Linux KVM hypervisor connected to the hrtimer subsystem, workqueue, or perf_event? The model knows. I asked ChatGPT and it claimed "all three". Any linux wizards who can confirm or deny? Anyway, in my experience using mainly the Claude chat to do some basic (not security) bug hunting, it usually fixates on one specific hypothesis, and it takes some effort to get it off that wrong track, even when I already k…

The wonderful thing though is that you can just run the model multiple times (even in parallel). Some instances might get stuck but as long as some find the bug and you have a good way to filter outputs (e.g. with another llm that tries to create concrete exploits) even a very small success rate on stage 1 can lead to reliable exploits

Re: Vulnerability research is cooked

#73

Post by GTP »

> The new models find real stuff. Forget the slop; will projects be able to keep up with a steady feed of verified, reproducible, reliably-exploitable sev:hi vulnerabilities?

If LLMs are as capable as said in the article, there will be an initial wave of security vulnerabilities. But then, all vulnerabilities will be discovered (or at least, LLMs will not find any more), and only new code will introduce new vulnerabilities. And everyone will be using LLMs to check the new code. So, regardless of what they say is correct or not, the problem doesn't really exist.

Re: Vulnerability research is cooked

#74

Post by charcircuit »

saintfire wrote: post #68
lll-o-lll wrote: post #58

Earlier quoted context omitted.

Eventual convergence? Assuming each defect fix has a 30% chance of introducing a new defect, we keep cycling until done?

Assuming you can catch every new bug it introduces. Both assumptions being unlikely. You also end up with a code base you let an AI agent trample until it is satisfied; ballooned in complexity and redudant brittle code.

You can have an AI agent refactor and improve code quality.

Re: Vulnerability research is cooked

#75

Post by m132 »

A hard read for a skeptic like me. A lot of speculation and extrapolation of a trend, not to say outright exaggeration, but very little actual data. Let's not forget that we're at the tip of an economic bubble, and what you're writing about is at the very center of it!

For what it's worth, I read Anthropic's write-up of their recent 0-day hunt that most of this post seems to be based on, and I can't help but notice that (assuming the documented cases were the most "spectacular") their current models mostly "pattern-matched" their ways towards the exploits; in all documented cases, the actual code analysis failed and the agents redeemed themselves by looking for known-vulnerable patterns they extracted from the change history or common language pitfalls. So, most of the findings, if not all, were results of rescanning the entire codebase for prior art. The corporate approach to security, just a little more automated.

Hence I agree with "the smartest vulnerability researcher" mentioned near the end. Yes, the most impactful vulnerabilities tend to be the boring ones, and catching those fast will make a big difference, but vulnerability research is far from cooked. If anything, it will get much more interesting.

Re: Vulnerability research is cooked

#76

Post by bsder »

reactordev wrote: post #46
Veserv wrote: post #19

Earlier quoted context omitted.

When did we enter the twilight zone where bug trackers are consistently empty? The limiting factor of bug reduction is remediation, not discovery. Even developer smoke testing usually surfaces bugs at a rate far faster than they can be fixed let alone actual QA. To be fair, the limiting factor in remediation is usually finding a reproducible test case which a vulnerability is by necessity. But, I would still bet most…

That might have been true pre LLMs but you can literally point an agent at the queue until it’s empty now.

The fact that KiCad still has a ton of highly upvoted missing features and the fact that FreeCAD still hasn't solved the topological renumbering problem are existence proofs to the contrary.

Re: Vulnerability research is cooked

#77

Post by miki123211 »

I think our saving grace is the fact that, especially in these hardened environments, it's a lot easier to find exploitable bugs than to actually exploit them.

As the defenders will have access to the same agents as the attackers, everybody will (mostly) find the same bugs. If recent trends continue[1], it's likely that major labs will make new models available to defenders first, making the attackers' jobs even harder.

What really worries me is models quickly developing exploits based on freshly-released patches, before most people had a chance to update. Big cloud vendors will likely have the ability to coordinate and deploy updates before the commits hit Github, smaller enterprise on-prem environments won't have that luxury.

Re: Vulnerability research is cooked

#78

Post by samuelknight »

gdulli wrote: post #62
samuelknight wrote: post #60

LLMs are expert hackers because: 1) They are expert coders, including a decently comprehensive CVE knowledge 2) They know every programming language/framework/stack 3) They know every human language They already have super human breadth and attention. And their depth is either super human or getting there. The state of the security industry through 2025 was expensive appsec human reviewers or primitive scanners. Now…

So much of the current internet is posts that read as a superposition of sincere and parody, and until that's resolved how do you know how to respond?

If that was a jab it my writing then yes, I am absolutely being sincere because I am an expert on this topic. LLMs went from being ok at one-shoting a function a to being so good at hacking that it's difficult to evaluate them. Prospective customers get back to us after a demo and tell us about the exploits it found on their services that are so vague and technical that they wouldn't think to look for them.

Re: Vulnerability research is cooked

#80

Post by rybosworld »

reactordev wrote: post #46
Veserv wrote: post #19

Earlier quoted context omitted.

When did we enter the twilight zone where bug trackers are consistently empty? The limiting factor of bug reduction is remediation, not discovery. Even developer smoke testing usually surfaces bugs at a rate far faster than they can be fixed let alone actual QA. To be fair, the limiting factor in remediation is usually finding a reproducible test case which a vulnerability is by necessity. But, I would still bet most…

That might have been true pre LLMs but you can literally point an agent at the queue until it’s empty now.

Shouldn't be down voted for saying this. There are active repo's this is happening in.

"BuT ThE LlM iS pRoBaBlY iNtRoDuCiNg MoRe BuGs ThAn It FiXeS"

This is an absurd take.

Post reply on HN