Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The coming industrialisation of exploit generation with LLMs

sean.heelan.io

Post reply on HN
Print view View original

101–110 of 174 posts

Re: The coming industrialisation of exploit generation with LLMs

#101

Post by raesene9 »

protocolture wrote: post #2

I genuinely dont know who to believe. The people who claim LLMs are writing excellent exploits. Or the people who claim that LLMs are sending useless bug reports. I dont feel like both can really be true.

Yeah they definitely can be true (IME), as there's a massive difference depending on how LLMs are used to the quality of the output.

For example if you just ask an LLM in a browser with no tool use to "find a vulnerability in this program", it'll likely give you something but it is very likely to be hallucinated or irrelevant.

However if you use the same LLM model via an agent, and provide it with concrete guidance on how to test its success, and the environment needed to prove that success, you are much more likely to get a good result.

It's like with Claude code, if you don't provide a test environment it will often make mistakes in the coding and tell you all is well, but if you provide a testing loop it'll iterate till it actually works.

Re: The coming industrialisation of exploit generation with LLMs

#102

Post by saagarjha »

cookiengineer wrote: post #84
tptacek wrote: post #55

Earlier quoted context omitted.

They're prioritizing memory corruption vulnerabilities, is the point of going to extremes to ensure there's no compiled C in their binaries.

It would be nice if there was something similar to the ebpf verifier, but for static C, so that loop mistakes, out of boundary mistakes and avoidable satisfiability problems are caught right in the compile step. The reason I'm so avoidant to using C libraries at all cost is that the ecosystem doesn't prioritize maintenance or other forms of code quality in its distribution. If you have to go to great lengths of havin…

What do you consider to be a loop mistake?

Re: The coming industrialisation of exploit generation with LLMs

#103

Post by saagarjha »

0xDEAFBEAD wrote: post #73
tptacek wrote: post #47

Earlier quoted context omitted.

"C executables" are most of the frontier of exploit development, which is why this is a meaningful model problem.

Can we fight fire with fire, and use LLMs to rewrite all the C in Rust?

Usually rewriting something in Rust requires nontrivial choices on the part of the translator that I’m not sure are currently within the reach of LLMs.

Re: The coming industrialisation of exploit generation with LLMs

#104

Post by saagarjha »

rightbyte wrote: post #99
NitpickLawyer wrote: post #93

Earlier quoted context omitted.

> These tools feel symmetric for defenders to use as well. I don't think so. From a pure mathematical standpoint, you'd need better (or equal) results at avg@1 or maj@x, while the attacker needs just pass@x to succeed. That is, the red agent needs to work just once, while the blue agent needs to work all the time. Current agents are much better (20-30%) at pass@x than maj@x. In real life that's why you sometimes see…

>> These tools feel symmetric for defenders to use as well. > I don't think so. From a pure mathematical standpoint, you'd need better (or equal) results at avg@1 or maj@x, while the attacker needs just pass@x to succeed. Executing remote code is a choice not some sort of force of nature. Timesharing systems are inherently not safe and way too much effort is put into claiming the stone from Sisyphus. SaaS and complex…

Awesome! What’s your strategy for migration of the entire world’s infrastructure to whatever you’re thinking about?

Re: The coming industrialisation of exploit generation with LLMs

#105

Post by saagarjha »

0xDEAFBEAD wrote: post #80
pizlonator wrote: post #79

Earlier quoted context omitted.

They don’t. For the calculus to change, anyone running an LLM to find bugs would have to be able to find all of the bugs that anyone else running an LLM could ever find. That’s not going to happen.

Correct me if I'm wrong, but I think a better mental model would be something like: Take the union of all bugs found by all white hats, fix all of those, then check if any black hat has found sufficient unfixed bugs to construct an exploit chain?

How do you check this?

Re: The coming industrialisation of exploit generation with LLMs

#106

Post by saagarjha »

cellis wrote: post #64
nl wrote: post #63

One of the interesting things to me about this is that Codex 5.2 found the most complex of the exploits. The reflects my experience too. Opus 4.5 is my everyday driver - I like using it. But Codex 5.2 with Extra High thinking is just a bit more powerful. Also despite what people say, I don't believe progress in LLM performance is slowing down at all - instead we are having more trouble generating tasks that are hard…

The “hard enough” tasks are all behind IP walls. If it’s a “hard enough” that generally means it’s a commercial problem likely involving disparate workflows and requiring a real human who probably isn’t a) inclined and/or b) permitted, to publish the task. The incentives are aligned to capture all value from solving that task as long as possible and only then publish.

I solve plenty of hard problems as a hobby

Re: The coming industrialisation of exploit generation with LLMs

#108

Post by rightbyte »

saagarjha wrote: post #104
rightbyte wrote: post #99

Earlier quoted context omitted.

>> These tools feel symmetric for defenders to use as well. > I don't think so. From a pure mathematical standpoint, you'd need better (or equal) results at avg@1 or maj@x, while the attacker needs just pass@x to succeed. Executing remote code is a choice not some sort of force of nature. Timesharing systems are inherently not safe and way too much effort is put into claiming the stone from Sisyphus. SaaS and complex…

Awesome! What’s your strategy for migration of the entire world’s infrastructure to whatever you’re thinking about?

My strategy is to not use "the entire world's infrastructure" which makes it redundant.

If enough people cancel their leftpad-as-a-Service subscription the server can be unplugged.

(Yes I am somewhat hyperbolic and yes I see use for internet connected servers and clients. I argue against the SaaS driven centralization.)

Re: The coming industrialisation of exploit generation with LLMs

#109

Post by saagarjha »

> The exploits generated do not demonstrate novel, generic breaks in any of the protection mechanisms. They take advantage of known flaws in those protection mechanisms and gaps that exist in real deployments of them. These are the same gaps that human exploit developers take advantage of, as they also typically do not come up with novel breaks of exploit mitigations for each exploit.

I actually think this result is a little disappointing but I largely chalk it up to the limited budget the author invested. In the CTF space we’re definitely seeing this more and more as models effectively “oneshot” typical pwn tasks that were significant effort to do by hand before. I feel like the pieces to do these are vaguely present in training data and the real constraints have been how fiddly and annoying they are to set up. An LLM is going to be well suited at this.

More interestingly, though, I suspect we will actually see software at least briefly get more secure as a result of this: I think a lot of incomplete implementations of mitigations are going to fall soon and (humans, for now) will be forced to keep up and patch them properly. This will drive investment in formal modeling of exploits, which is currently a very immature field.

Re: The coming industrialisation of exploit generation with LLMs

#110

Post by prodigycorp »

nl wrote: post #72
conception wrote: post #66

Earlier quoted context omitted.

The Anthropic models are great workers/tool users. OpenAI Codex High is a great reviewer/fixer. Gemini is the genius repainting your bathroom walls into a Monet from memory because you mentioned once a few weeks ago you liked classical art and needed to repaint your bathroom. Gemini didn’t mention the task or that it was starting it. It did a pretty good job after you had to admit.

Disagree about Codex - it's great at doing things too! Gemini either does a Monet or demolishes your bathroom and builds a new tuna fishing boat there instead, and it is completely random which one you get. It's a great model but I rarely use it because it's so random as to what you get.

gpt models are crazy good. They just take forever.
Post reply on HN