Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The coming industrialisation of exploit generation with LLMs

sean.heelan.io

Post reply on HN
Print view View original

51–60 of 174 posts

Re: The coming industrialisation of exploit generation with LLMs

#51

Post by pixl97 »

hackyhacky wrote: post #13
er4hn wrote: post #3

I think the author makes some interesting points, but I'm not that worried about this. These tools feel symmetric for defenders to use as well. There's an easy to see path that involves running "LLM Red Teams" in CI before merging code or major releases. The fact that it's a somewhat time expensive (I'm ignoring cost here on purpose) test makes it feel similar to fuzzing for where it would fit in a pipeline. New tool…

> I think the author makes some interesting points, but I'm not that worried about this. Given the large number of unmaintained or non-recent software out there, I think being worried is the right approach. The only guaranteed winner is the LLM companies, who get to sell tokens to both sides.

I mean you're leaving out large nation state entities

Re: The coming industrialisation of exploit generation with LLMs

#52

Post by pixl97 »

airza wrote: post #19
rwmj wrote: post #12

Earlier quoted context omitted.

With the exploits, you can try them and they either work or they don't. An attacker is not especially interested in analysing why the successful ones work. With the CVE reports some poor maintainer has to go through and triage them, which is far more work, and very asymmetrical because the reporters can generate their spam reports in volume while each one requires detailed analysis.

All the attackers I’ve known are extremely, pathologically interested in understanding why their exploits work.

Very often they need to understand it well to chain exploits

Re: The coming industrialisation of exploit generation with LLMs

#53

Post by er4hn »

amelius wrote: post #24
er4hn wrote: post #3

I think the author makes some interesting points, but I'm not that worried about this. These tools feel symmetric for defenders to use as well. There's an easy to see path that involves running "LLM Red Teams" in CI before merging code or major releases. The fact that it's a somewhat time expensive (I'm ignoring cost here on purpose) test makes it feel similar to fuzzing for where it would fit in a pipeline. New tool…

> These tools feel symmetric for defenders to use as well. Why? The attackers can run the defending software as well. As such they can test millions of testcases, and if one breaks through the defenses they can make it go live.

Right, that's the same situation as fuzz testing today, which is why I compared it. I feel like you're gesturing towards "Attackers only need to get lucky once, defenders need to do a good job everytime" but a lot of the times when you apply techniques like fuzz testing it doesn't take a lot of effort to get good coverage. I suspect a similar situation will play out with LLM assisted attack generation. For higher value targets based on OSS, there's projects like Google Big Sleep to bring enhanced resources.

Re: The coming industrialisation of exploit generation with LLMs

#54

Post by akoboldfrying »

cookiengineer wrote: post #50
akoboldfrying wrote: post #46

Earlier quoted context omitted.

Why would statically linking a library reduce the number of vulnerabilities in it? AFAICT, static linking just means the set of vulnerabilities you get landed with won't change over time.

> Why would statically linking a library reduce the number of vulnerabilities in it? I use pure go implementations only, and that implies that there's no statically linked C ABI in my binaries. That's what disabling CGO means.

What I mean is: There will be bugs* in that pure Go implementation, and static linking means you're baking them in forever. Why is this preferable to dynamic linking?

* It's likely that C implementations will have bugs related to dynamic memory allocation that are absent from the Go implementation, because Go is GCed while C is not. But it would be very surprising if there were no bugs at all in the Go implementation.

Re: The coming industrialisation of exploit generation with LLMs

#55

Post by tptacek »

akoboldfrying wrote: post #54
cookiengineer wrote: post #50

Earlier quoted context omitted.

> Why would statically linking a library reduce the number of vulnerabilities in it? I use pure go implementations only, and that implies that there's no statically linked C ABI in my binaries. That's what disabling CGO means.

What I mean is: There will be bugs* in that pure Go implementation, and static linking means you're baking them in forever. Why is this preferable to dynamic linking? * It's likely that C implementations will have bugs related to dynamic memory allocation that are absent from the Go implementation, because Go is GCed while C is not. But it would be very surprising if there were no bugs at all in the Go implementation…

They're prioritizing memory corruption vulnerabilities, is the point of going to extremes to ensure there's no compiled C in their binaries.

Re: The coming industrialisation of exploit generation with LLMs

#56

Post by AdieuToLogic »

protocolture wrote: post #2

I genuinely dont know who to believe. The people who claim LLMs are writing excellent exploits. Or the people who claim that LLMs are sending useless bug reports. I dont feel like both can really be true.

Both can be true if each group selectively provides LLM output supporting their position. Essentially, this situation can be thought of as a form of the Infinite Monkey Theorem[0] where the result space is drastically reduced from "purely random" to "likely to be statistically relevant."

For an interesting overview of the above theorem, see here[1].

0 - https://en.wikipedia.org/wiki/Infinite_monkey_theorem

1 - https://www.yalescientific.org/2025/04/sorry-shakespeare-why...

Re: The coming industrialisation of exploit generation with LLMs

#57

Post by Vetch »

digdugdirk wrote: post #38
er4hn wrote: post #3

I think the author makes some interesting points, but I'm not that worried about this. These tools feel symmetric for defenders to use as well. There's an easy to see path that involves running "LLM Red Teams" in CI before merging code or major releases. The fact that it's a somewhat time expensive (I'm ignoring cost here on purpose) test makes it feel similar to fuzzing for where it would fit in a pipeline. New tool…

That's not how complex systems work though? You say that these tools feel "symmetric" for defenders to use, but having both sides use the same tools immediately puts the defenders at a disadvantage in the "asymmetric warfare" context. The defensive side needs everything to go right, all the time. The offensive side only needs something to go wrong once.

I'm not sure that's the fully right mental model to use. They're not searching randomly with unbounded compute nor selecting from arbitrary strategies in this example. They are both using LLMs and likely the same ones, so will likely uncover overlapping possible solutions. Avoiding that depends on exploring more of the tail of the highly correlated to possibly identical distributions.

It's a subtle difference from what you said in that it's not like everything has to go right in a sequence for the defensive side, defenders just have to hope they committed enough into searching such that the offensive side has a significantly lowered chance of finding solutions they did not. Both the attackers and defenders are attacking a target program and sampling the same distribution for attacks, it's just that the defender is also iterating on patching any found exploits until their budget is exhausted.

Re: The coming industrialisation of exploit generation with LLMs

#58

Post by wat10000 »

SchemaLoad wrote: post #14
rwmj wrote: post #12

Earlier quoted context omitted.

With the exploits, you can try them and they either work or they don't. An attacker is not especially interested in analysing why the successful ones work. With the CVE reports some poor maintainer has to go through and triage them, which is far more work, and very asymmetrical because the reporters can generate their spam reports in volume while each one requires detailed analysis.

There's been several notable posts where maintainers found there was no bug at all, or the example code did not even call code from their project and had just found running a python script can do things on your computer. Entirely AI generated Issue reports and examples wasting maintainer time.

I've had multiple reports with elaborate proofs of concept that boil down to things like calling dlopen() on a path to a malicious library and saying dlopen has a security vulnerability.

Re: The coming industrialisation of exploit generation with LLMs

#59

Post by wat10000 »

protocolture wrote: post #2

I genuinely dont know who to believe. The people who claim LLMs are writing excellent exploits. Or the people who claim that LLMs are sending useless bug reports. I dont feel like both can really be true.

LLMs produce good output and bad output. The trick is figuring out which is which. They excel at tasks where good output is easily distinguished. For example, I've had a lot of success with making small reproducers for bugs. I see weird behavior A coming from giant pile of code B, figure out how to trigger A in a small example. It can often do so, and when it gets it wrong it's easy to detect because its example doesn't actually do A. The people sending useless bug reports aren't checking for good output.

Re: The coming industrialisation of exploit generation with LLMs

#60

Post by socketcluster »

The continuous lowering of entry barriers to software creation, combined with the continuous lowering of entry barriers to software hacking is an explosive combination.

We need new platforms which provide the necessary security guardrails, verifiability, simplicity of development, succinctness of logic (high feature/code ratio)... You can't trust non-technical vibe coders with today's software tools when they can't even trust themselves.

Post reply on HN