Live data from Hacker News

The coming industrialisation of exploit generation with LLMs

sean.heelan.io

121–130 of 174 posts

Re: The coming industrialisation of exploit generation with LLMs

#121
post #5

> In the hardest task I challenged GPT-5.2 it to figure out how to write a specified string to a specified path on disk, while the following protections were enabled: address space layout randomisation, non-executable memory, full RELRO, fine-grained CFI on the QuickJS binary, hardware-enforced shadow-stack, a seccomp sandbox to prevent shell execution, and a build of QuickJS where I had stripped all functionality in…

Maybe we can remove mitigations. Every exploit you see is: First, find a vulnerability (the difficult part). Then, drill through five layers of ultimately ineffective "mitigations" (the tedious but almost always doable part). Probabilistic mitigations work against probabilistic attacks, I guess - but exploit writers aren't random, they are directed, and they find the weaknesses.

The vulnerability was found by Opus:

"This is true by definition as the QuickJS vulnerability was previously unknown until I found it (or, more correctly: my Opus 4.5 vulnerability discovery agent found it)."

Re: The coming industrialisation of exploit generation with LLMs

#122

Earlier quoted context omitted.

Correct me if I'm wrong, but I think a better mental model would be something like: Take the union of all bugs found by all white hats, fix all of those, then check if any black hat has found sufficient unfixed bugs to construct an exploit chain?

How do you check this?

I meant in the sense that this algorithm will tell you if your software is vulnerable in the abstract. It's not a procedure you could actually follow.

Re: The coming industrialisation of exploit generation with LLMs

#123

>Recently I ran an experiment where I built agents on top of Opus 4.5 and GPT-5.2 and then challenged them to write exploits for a zeroday vulnerability in the QuickJS Javascript interpreter. I think the main challenge for hackers is to find 0day vulnerabilities, not writing the actual exploit code.

The vulnerability was found by Claude:

>This is true by definition as the QuickJS vulnerability was previously unknown until I found it (or, more correctly: my Opus 4.5 vulnerability discovery agent found it).

Re: The coming industrialisation of exploit generation with LLMs

#124
two points -

1) it becomes increasingly more dangerous to dl stuff from the internet and just run it, even its opensource, given normally people don't read all of it. for weird repos I'd recomment to do automated analysis with opus 4.5 or the gpt 5.2 indeed.

2) if we assume adversaries are using LLMs to churn exploits 24/7, which we should absolutely do, perhaps the time where we turn the internet off whenever is not needed, is not far.

Re: The coming industrialisation of exploit generation with LLMs

#125
post #121

Earlier quoted context omitted.

Maybe we can remove mitigations. Every exploit you see is: First, find a vulnerability (the difficult part). Then, drill through five layers of ultimately ineffective "mitigations" (the tedious but almost always doable part). Probabilistic mitigations work against probabilistic attacks, I guess - but exploit writers aren't random, they are directed, and they find the weaknesses.

The vulnerability was found by Opus: "This is true by definition as the QuickJS vulnerability was previously unknown until I found it (or, more correctly: my Opus 4.5 vulnerability discovery agent found it)."

Makes little difference, whoever or whatever finds the initial exploit will also do the busywork of working around mitigations. (Techniques to work around mitigations are initially not busywork, but as soon as somehow has found a working principle, it seems to me that it becomes busywork)

Re: The coming industrialisation of exploit generation with LLMs

#126
post #124

two points - 1) it becomes increasingly more dangerous to dl stuff from the internet and just run it, even its opensource, given normally people don't read all of it. for weird repos I'd recomment to do automated analysis with opus 4.5 or the gpt 5.2 indeed. 2) if we assume adversaries are using LLMs to churn exploits 24/7, which we should absolutely do, perhaps the time where we turn the internet off whenever is not…

...well, just dont download random stuff from the internet and run it on your important machines then? :-))

You are right: 30 years ago, it was safe to go to vendor XY page and download his latest version and it was more or less waterproof. Today with all these mirror sites, very often better SEO ranking than the original, its quite dangerous: In my former bank we had a colleague who installed a browser add-in that he used for years (at home and in the bank); then he got a new notebook, fresh browser, he installed the same extension - but from a different source than the original vendor: unfortunately, this version contained malware and a big transaction was caught by compliance in the very last second, because he wasnt aware of data leakage.

Re: The coming industrialisation of exploit generation with LLMs

#127

Earlier quoted context omitted.

Not symmetric at all. There are countless bugs to fund. If the offender runs these tools, then any bug they find becomes a cyberweapon. If the defender runs these tools, they will not thwart the offender unless they find and fix all of the bugs. Any vs all is not symmetric

LLMs effectively move us from A to B: A) 1 cyber security employee, 1 determined attacker B) 100 cyber security employees, 100 determined attackers Which is better for defender?

Neither

Re: The coming industrialisation of exploit generation with LLMs

#128

Earlier quoted context omitted.

They don’t. For the calculus to change, anyone running an LLM to find bugs would have to be able to find all of the bugs that anyone else running an LLM could ever find. That’s not going to happen.

Correct me if I'm wrong, but I think a better mental model would be something like: Take the union of all bugs found by all white hats, fix all of those, then check if any black hat has found sufficient unfixed bugs to construct an exploit chain?

The black hat has to find a handful of bugs. Sometimes one bug is enough.

Re: The coming industrialisation of exploit generation with LLMs

#129
post #115
post #5

> In the hardest task I challenged GPT-5.2 it to figure out how to write a specified string to a specified path on disk, while the following protections were enabled: address space layout randomisation, non-executable memory, full RELRO, fine-grained CFI on the QuickJS binary, hardware-enforced shadow-stack, a seccomp sandbox to prevent shell execution, and a build of QuickJS where I had stripped all functionality in…

So much for ‘stochastic parrots’

> The exploits generated do not demonstrate novel, generic breaks in any of the protection mechanisms.

Re: The coming industrialisation of exploit generation with LLMs

#130

Earlier quoted context omitted.

> Why would statically linking a library reduce the number of vulnerabilities in it? I use pure go implementations only, and that implies that there's no statically linked C ABI in my binaries. That's what disabling CGO means.

What I mean is: There will be bugs* in that pure Go implementation, and static linking means you're baking them in forever. Why is this preferable to dynamic linking? * It's likely that C implementations will have bugs related to dynamic memory allocation that are absent from the Go implementation, because Go is GCed while C is not. But it would be very surprising if there were no bugs at all in the Go implementation…

About a year ago I had some code I had been working on for about a year subject to a pretty heavy-duty security review by a reputable review company. When they asked what language I implemented it in and I told them "Go", they joked that half their job was done right there.

While Go isn't perfect and you can certainly write some logic bugs that sufficiently clever use of a more strongly-typed language might let you avoid (though don't underestimate what sufficiently clever use of what Go already has can do for you either when wielded with skill), it has a number of characteristics that keep it somewhat safer than a lot of other languages.

First, it's memory safe in general, which obviously out of the gate helps a lot. You can argue about some super, super fringe cases with unprotected concurrent access to maps, but you're still definitely talking about something on the order of .1% to .01% of the surface area of C.

Next, many of the things that people complain about Go on Hacker News actually contribute to general safety in the code. One of the biggest ones is that it lacks any ability to take an string and simply convert it to a type, which has been the source of catastrophic vulnerabilities in Ruby [1] and Java (Log4Shell), among others. While I use this general technique quite frequently, you have to build your own mechanism for it (not a big deal, we're talking ~50 lines of code or so tops) and that mechanism won't be able to use any class (using general terminology, Go doesn't have "classes" but user-defined types fill in here) that wasn't explicitly registered, which sharply contains the blast radius of any exploit. Plus a lot of the exploits come from excessively clever encoding of the class names; generally when I simply name them and simply do a single lookup in a single map there isn't a lot of exploit wiggle room.

In general though it lacks a lot of the features that get people in trouble that aren't related to memory unsafety. Dynamic languages as a class start out behind the eight-ball on this front because all that dynamicness makes it difficult to tell exactly what some code might do with some input; goodness help you if there's a path to the local equivalent of "eval".

Go isn't entirely unique in this. Rust largely shares the same characteristics, there's some others that may qualify. But some other languages you might expect to don't; for instance, at least until recently Java had a serious problem with being able to get references to arbitrary classes via strings, leading to Log4Shell, even though Java is a static language. (I believe they've fixed that since then but a lot of code still has to have the flag to flip that feature back on because they depend on it in some fundamental libraries quite often.) Go turns out to be a relatively safe security language to write in compared to the landscape of general programming languages in common use. I add "in common use" and highlight it here because I don't think it's anywhere near optimal in the general landscape of languages that exist, nor the landscape of languages that ought to exist and don't yet. For instance in the latter case I'd expect capabilities to be built in to the lowest layer of a language, which would further do great, great damage to the ability to exploit such code. However no such language is in common use at this time. Pragmatically when I need to write something very secure today, Go is surprisingly high on my short list; theoretically I'm quite dissatisfied.

[1]: https://blog.trailofbits.com/2025/08/20/marshal-madness-a-br...

Post reply on HN