Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vulnerability research is cooked

sockpuppet.org

Post reply on HN
Print view View original

91–100 of 181 posts

Re: Vulnerability research is cooked

#91

Post by habinero »

lifty wrote: post #29
tptacek wrote: post #12

Earlier quoted context omitted.

That might be one outcome, especially for large, expertly-staffed vendors who are already on top of this stuff. My real interest in what happens to the field for vulnerability researchers.

Perhaps a meta evolution, they become experts at writing harnesses and prompts for discovering and patching vulnerabilities in existing code and software. My main interest is, now that we have LLMs, will the software industry move to adopting techniques like formal verification and other perhaps more lax approaches that massively increase the quality of software.

Testing exists.

> formal verification

Outside of limited specific circumstances, formal verification gives you nothing that tests don't give you, and it makes development slow and iteration a chore. People know about it, and it's not used for lot of reasons.

Re: Vulnerability research is cooked

#92

Post by tptacek »

joatmon-snoo wrote: post #90
tptacek wrote: post #51

Earlier quoted context omitted.

People have said that for decades and it wasn't true until recently.

Hmm: can you elaborate? I've never been on a security-specific team, but it's always seemed to me that triggering a bug is, for the median issue, easier than fixing it, and I mentally extend that to security issues. This holds especially true if the "bug" is a question about "what is the correct behavior?", where the "current behavior of the system" is some emergent / underspecified consequence of how different featu…

It has generally been the case that (1) finding and (2) reliably exploiting vulnerabilities is much more difficult than patching them. In fact, patching them is often so straightforward that you can kill whole bug subspecies just by sweeping the codebase for the same pattern once you see a bug. You'd do that just sort of as a matter of course, without necessarily even qualifying the bugs you're squashing are exploitable.

As bugs get more complicated, that asymmetry has become less pronounced, but the complexity of the bugs (and their patches) is offset by the increased difficulty of exploiting them, which has become an art all its own.

LLMs sharply tilt that difficulty back to the defender.

Re: Vulnerability research is cooked

#93

Post by MeetingsBrowser »

Legend2440 wrote: post #34
spr-alex wrote: post #10

I interned for the author at 18. I assumed security testing worked like this: 1. Static analysis catches nearly all bugs with near-total code coverage 2. Private tooling extends that coverage further with better static analysis and dynamic analysis, and that edge is what makes contractors valuable 3. Humans focus on design flaws and weird hardware bugs like cryptographic side-channels from electromagnetic emanations…

Catching all bugs with static analysis would involve solving the halting problem, so it's never going to happen.

Catching all bugs with static analysis is actually really easy, as long as you don't mind false positives.

Re: Vulnerability research is cooked

#94

Post by cvwright »

stavros wrote: post #8

I don't understand why the takeaway here is (unless I'm missing something), more or less "everything is going to get exploited all the time". If LLMs can really find a ton of vulnerabilities in my software, why would I not run them and just patch all the vulnerabilities, leading to perfectly secure software (or, at the very least, software for which LLMs can no longer find any new vulnerabilities)?

Find-then-patch only works if you can fix the bugs quicker than you’re creating new ones.

Some orgs will be able to do this, some won’t.

Re: Vulnerability research is cooked

#95

Post by dumpsterdiver »

vibe42 wrote: post #66

If everyone is running the same models, does this not favour white hat / defense? Since many exploits consists of several vulnerabilities used in a chain, if a LLM finds one in the middle and it's fixed, that can change a zero day to something of more moderate severity? E.g. someone finds a zero day that's using three vulns through different layers. The first and third are super hard to find, but the second is of mod…

> If everyone is running the same models, does this not favour white hat / defense?

The landscape is turbulent (so this comment might be outdated by the time I submit it), but one thing I’m catching between the lines is a resistance to provide defensive coding patterns because (guessing) they make the flaw they’re defending against obvious. When the flaw is widespread - those patterns effectively make it cheap to attack for observant eyes.

After seeing the enhanced capabilities recently, my conspiracy theory is that models do indeed traverse the pathways containing ideal mitigations, but they fall back to common anti-patterns when they hit the guardrails. Some of the things I’ve seen are baffling, and registered as adversarial on my radar.

Re: Vulnerability research is cooked

#96

Post by thadt »

love2read wrote: post #70
thadt wrote: post #44

So the intersting question: are we long term safer with "simpler" closer to hardware memory unsafe(ish) environments like Zig, or is the memory safe but more abstract feature set of languages like Rust still the winning direction? If a hypothetical build step is "look over this program and carfully examine the bounds of safety using your deep knowledge of the OS, hardware, language and all the tools that come along w…

What about this article raises this question? If anything, this article makes it pretty clear that memory safe languages are a win. It seems like a serious disadvantage to require a nondeterministic program to evaluate your code's safety.

In general I agree and suspect that memory safety is a tool that will continue to pay dividends for some time.

But there are tradeoffs and more ways to write correct and 'safe' code than doing it in a "memory safe" language. If frontier models indeed are a step function in finding vulnerabilities, then they're also a step function in writing safer code. We've been able to write safety critical C code with comprehensive testing for a long time (with SQLite presenting a well known critique of the tradeoffs).

The rub has been that writing full coverage tests, fuzzing, auditing, etc. has been costly. If those costs have changed, then it's an interesting topic to try to undertand how.

Re: Vulnerability research is cooked

#97

Post by tptacek »

MeetingsBrowser wrote: post #93
Legend2440 wrote: post #34

Earlier quoted context omitted.

Catching all bugs with static analysis would involve solving the halting problem, so it's never going to happen.

Catching all bugs with static analysis is actually really easy, as long as you don't mind false positives.

Conventional static analysis tools come nowhere close to catching all bugs, even accounting for the false positives.

Re: Vulnerability research is cooked

#98

Post by m132 »

acdha wrote: post #82
m132 wrote: post #75

A hard read for a skeptic like me. A lot of speculation and extrapolation of a trend, not to say outright exaggeration, but very little actual data. Let's not forget that we're at the tip of an economic bubble, and what you're writing about is at the very center of it! For what it's worth, I read Anthropic's write-up of their recent 0-day hunt that most of this post seems to be based on, and I can't help but notice t…

I tend to be skeptical but listening to the linked podcast with Carlini and found him very credible–not a sales guy, not an AI doomer, but someone talking about how little work he had to do to find real exploits in heavily-fuzzed code. I think there’s still a safe bet that many apps will be cumbersome to attack but I think it’s still going to happen faster than I used to think. https://securitycryptographywhatever.co…

Thanks. Watched most of this talk and, unless I missed something, it seems to confirm what I was thinking—most of the strength currently comes from the scale you can deploy LLMs at, not them being better at vulnerability research than humans (if you factor out the throughput). And since this is a relatively new development, nobody really knows right now if this is going to have a greater impact than fuzzers and static analyzers had, or if newer models are ever going to get to a level that'd make computer security a solved problem.

Re: Vulnerability research is cooked

#99

Post by stavros »

cvwright wrote: post #94
stavros wrote: post #8

I don't understand why the takeaway here is (unless I'm missing something), more or less "everything is going to get exploited all the time". If LLMs can really find a ton of vulnerabilities in my software, why would I not run them and just patch all the vulnerabilities, leading to perfectly secure software (or, at the very least, software for which LLMs can no longer find any new vulnerabilities)?

Find-then-patch only works if you can fix the bugs quicker than you’re creating new ones. Some orgs will be able to do this, some won’t.

"Find me vulnerabilities in this PR."

Re: Vulnerability research is cooked

#100

Post by array_key_first »

rybosworld wrote: post #80
reactordev wrote: post #46

Earlier quoted context omitted.

That might have been true pre LLMs but you can literally point an agent at the queue until it’s empty now.

Shouldn't be down voted for saying this. There are active repo's this is happening in. "BuT ThE LlM iS pRoBaBlY iNtRoDuCiNg MoRe BuGs ThAn It FiXeS" This is an absurd take.

It probably is introducing more bugs because I think some people dont understand how bugs work.

Very, very rarely is a bug a mistake. As in, something unintentional that you just fix and boom, done.

No no. Most bugs are intentional, and the bug part is some unintended side effects that is a necessary, but unforseen, consequence of the main effect. So, you can't just "fix" the bug without changing behavior, changing your API, changing garauntees, whatever.

And that's how you get the 1 month 1-liner. Writing the one line is easy. But you have to spend a month debating if you should do it, and what will happen if you do.

Post reply on HN