Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vulnerability research is cooked

sockpuppet.org

Post reply on HN
Print view View original

101–110 of 181 posts

Re: Vulnerability research is cooked

#101

Post by acdha »

tptacek wrote: post #83
acdha wrote: post #82

Earlier quoted context omitted.

I tend to be skeptical but listening to the linked podcast with Carlini and found him very credible–not a sales guy, not an AI doomer, but someone talking about how little work he had to do to find real exploits in heavily-fuzzed code. I think there’s still a safe bet that many apps will be cumbersome to attack but I think it’s still going to happen faster than I used to think. https://securitycryptographywhatever.co…

Nicholas Carlini is the real deal. He was most recently on the front page for "How to win a best paper award", about his experience winning a series of awards at Big 4 academic security conferences, mostly recently for work he coauthored with Adi Shamir (I'm just namedropping the obvious name) on stealing the weights from deep neural networks. Before all that (and before he got his doctorate), he and Hans Nielsen wro…

Thanks for having him on. It was really nice to hear a sober, experienced voice talking about their work with fellow practitioners.

Re: Vulnerability research is cooked

#102

Post by nickpsecurity »

lifty wrote: post #29
tptacek wrote: post #12

Earlier quoted context omitted.

That might be one outcome, especially for large, expertly-staffed vendors who are already on top of this stuff. My real interest in what happens to the field for vulnerability researchers.

Perhaps a meta evolution, they become experts at writing harnesses and prompts for discovering and patching vulnerabilities in existing code and software. My main interest is, now that we have LLMs, will the software industry move to adopting techniques like formal verification and other perhaps more lax approaches that massively increase the quality of software.

Also, synthetic data and templates to help them discover new vulnerabilities or make agents work on things they're bad at. They differentiate with their prompts or specialist models.

Also, like ForAllSecure's Mayhem, I think they can differentiate on automatic patching that's reliable and secure. Maybe test generation, too, that does full coverage. They become drive by verification and validation specialists who also fix your stuff for you.

Re: Vulnerability research is cooked

#103

Post by nickpsecurity »

spr-alex wrote: post #86
IsTom wrote: post #61

Earlier quoted context omitted.

A lot of software doing useful work halts pretty trivialy, consuming inputs and doing bounded computation on each of them. You're not going to recurse much in click handlers or keep making larger requests to handle the current one.

I was just very naive at 18 about program analysis. I haven't lost my imagination though. I was a self-taught IOI gold division competitor. I thought every problem had an algorithm. It doesn't work like that. Program analysis is collecting special snowflakes that melt in your hand. There is no end to the ways you can write a bug in C. Ghosts of Semmle, Semgrep, Coccinelle past, be humbled. LLMs saturate test coverage…

You've never seen the full power of static analysis, dynamic analysis, and test generation. The best examples were always silo'd, academic codebases. If they were combined, and matured, the results would be amazing. I wanted to do that back when I was in INFOSEC.

That doesn't even account for lightweight, formal methods. SPARK Ada, Jahob verification system with its many solvers, Design ny Contract, LLM's spitting this stuff out from human descriptions, type systems like Rust's, etc. Speed run (AI) producing those with unsafe stuff checked by the combo of tools I already described.

Re: Vulnerability research is cooked

#104

Post by nickpsecurity »

thadt wrote: post #44

So the intersting question: are we long term safer with "simpler" closer to hardware memory unsafe(ish) environments like Zig, or is the memory safe but more abstract feature set of languages like Rust still the winning direction? If a hypothetical build step is "look over this program and carfully examine the bounds of safety using your deep knowledge of the OS, hardware, language and all the tools that come along w…

They're great at Python and Javascript which have lots of tooling. My idea was to make X-to-safe-lang translators, X initially being Python and Javascript. Let the tools keep generating what they're good at. The simpler translators make it safe and fast.

If translated to C or Java, we can use decades worth of tools for static analysis and test generation. While in Python and Javascript, it's easier to analyze and live debug by humans.

Multiple wins if the translators can be built.

Re: Vulnerability research is cooked

#105

Post by ActorNightly »

somesortofthing wrote: post #81

Am I wrong in thinking that an "exploits are free" environment massively favors the defender? Given that real-world exploits usually chain 0days, the attacker has to find the whole chain while the defender only needs to fix the weakest link. The defender also gets to make the first move by just putting a "run an agent to find vulns" step in their CI pipeline. If LLMs truly make finding exploits free, almost no LLM-fi…

You aren't wrong for the most part, but this whole thing of "find me vulns" is not really accurately describing the space.

Finding vulns has almost become sort of like a vibe thing even before LLMs. There would be some security patch that everyone says is critical because it fixes a vulnerability, but the vulnerability is like "under certain conditions, and given physical access to the device, an attacker can craft a special input that crashes the service"... and thats it.

Even stuff like Spectre and Meltdown, which I highly doubt an LLM can find on its own without specifically knowing about speculative execution attacks, are incredibly hard to use. People made a big deal of those being able to be used from javascript, but to actually leak anything of importance you need to know memory layouts, a bunch of other info and so on.

So while an LLM can patch all the up front vulnerabilities, most if not all of those are completely useless to an attacker. Modern systems are incredibly secure.

On the flip side, the stuff that LLM doesn't know about, that can be exploited. For example, assume that log4shell hasn't been found yet, and that log statements by default can pull jni objects from the internet and execute them. The llms would happily write you code with log statements using log4j, and pass it through vulnerability checker, and I would bet that even at the bytecode level it won't ever figure out that vulnerability exists.

And overall, because of Rice theorem, you can't tell if the program is fully exploitable or not without actually running it in some form and way. LLMS can help you with this (but not fully of course) by actually running it and fuzzing inputs and observing memory traces, but even this gets very hard when you introduce things like threading and timed executions which can all affect the result.

And also, the LLMs themselves are an exploit vector now. If you manage to intercept the API calls somehow and insert code or other instruction, you can have the developer essentially put the exploit for you into the code.

So I would say the field is about even.

Re: Vulnerability research is cooked

#106

Post by tptacek »

acdha wrote: post #101
tptacek wrote: post #83

Earlier quoted context omitted.

Nicholas Carlini is the real deal. He was most recently on the front page for "How to win a best paper award", about his experience winning a series of awards at Big 4 academic security conferences, mostly recently for work he coauthored with Adi Shamir (I'm just namedropping the obvious name) on stealing the weights from deep neural networks. Before all that (and before he got his doctorate), he and Hans Nielsen wro…

Thanks for having him on. It was really nice to hear a sober, experienced voice talking about their work with fellow practitioners.

Thank Nicholas! We'll talk to anyone. :)

Re: Vulnerability research is cooked

#107

Post by intended »

charcircuit wrote: post #74
saintfire wrote: post #68

Earlier quoted context omitted.

Assuming you can catch every new bug it introduces. Both assumptions being unlikely. You also end up with a code base you let an AI agent trample until it is satisfied; ballooned in complexity and redudant brittle code.

You can have an AI agent refactor and improve code quality.

It’s agents all the way down - until you have liability. At some point, it’s going to be someone’s neck on the line, and saying “the agents know” isn’t going to satisfy customers (or in a worst case, courts).

Re: Vulnerability research is cooked

#108

Post by [deleted] »

somesortofthing wrote: post #81

Am I wrong in thinking that an "exploits are free" environment massively favors the defender? Given that real-world exploits usually chain 0days, the attacker has to find the whole chain while the defender only needs to fix the weakest link. The defender also gets to make the first move by just putting a "run an agent to find vulns" step in their CI pipeline. If LLMs truly make finding exploits free, almost no LLM-fi…

[deleted]

Re: Vulnerability research is cooked

#109

Post by worthless-trash »

k_roy wrote: post #56
batshit_beaver wrote: post #52

Earlier quoted context omitted.

You literally cannot, since ANY changes to code tend to introduce unintended (or at least not explicitly requested) new behaviors.

Which still means a single person with Claude can clear a queue in a day versus a month with a traditional team.

Your example must have incredible users or really trivial software.

Re: Vulnerability research is cooked

#110

Post by gsnedders »

tptacek wrote: post #36
narginal wrote: post #32

Just like how fuzzers will find all the bugs, right? Right?? There's definitely infrastructure at these big companies that isn't sitting in a while loop 'fuzzing' right? Why is it news that vulnerability research will continue to get harder, exactly? It has always been this way, exploits will get more expensive, and the best researchers will continue with whatever tools they find useful.

It's a good question. Fuzzers generated a surge of new vulnerabilities, especially after institutional fuzzing clusters got stood up, and after we converged on coverage-guided fuzzers like AFL. We then got to a stable equilibrium, a new floor, such that vulnerability research & discovery doesn't look that drastically different after fuzzing as before. Two things to notice: * First, fuzzers also generated and continue…

The other thing is to remember is that when it comes to complex targets, attacks still are found by using a different fuzzer and/or targeting a different entry point.

It stands to reason that the same will apply for LLMs.

Post reply on HN