The coming industrialisation of exploit generation with LLMs • Hacker News Board

Re: The coming industrialisation of exploit generation with LLMs

#141

Post by titzer » Tue, Jan 20, 2026, 6:25 PM UTC

simonw wrote: post #5

> In the hardest task I challenged GPT-5.2 it to figure out how to write a specified string to a specified path on disk, while the following protections were enabled: address space layout randomisation, non-executable memory, full RELRO, fine-grained CFI on the QuickJS binary, hardware-enforced shadow-stack, a seccomp sandbox to prevent shell execution, and a build of QuickJS where I had stripped all functionality in…

There are so many holes at the bottom of the machine code stack. In the future we'll question why we didn't move to WASM as the universal executable format sooner. Instead, we'll try a dozen incomplete hardware mitigations first to try to mitigate backwards crap like overwriting the execution stack.

Re: The coming industrialisation of exploit generation with LLMs

#142

Post by GrinningFool » Tue, Jan 20, 2026, 6:37 PM UTC

NitpickLawyer wrote: post #94

SchemaLoad wrote: post #14

Earlier quoted context omitted.

There's been several notable posts where maintainers found there was no bug at all, or the example code did not even call code from their project and had just found running a python script can do things on your computer. Entirely AI generated Issue reports and examples wasting maintainer time.

That's because the user of the tool didn't go through the troubles to setup the env properly (as the author of the blog did). So what they got was a "story about a bug", but without verification. The proper way to use these tools (like in other verifiable tasks such as math or coding) is to give them a feedback loop and an easily verifiable success criteria. In security exploitation you either capture the flag or not…

Now all that's left is to get every person who uses them to generate bug reports to just follow these practices.

Re: The coming industrialisation of exploit generation with LLMs

#143

Post by pnathan » Tue, Jan 20, 2026, 6:56 PM UTC

KellyCriterion wrote: post #135

pnathan wrote: post #134

Earlier quoted context omitted.

> 30 years ago, it was safe to go to vendor XY page and download his latest version and it was more or less waterproof. You _are_ joking, right? I distinctly remember all sorts of dubious freewarez sites with slightly modified installers. 1997-2000 era. And anti-virus was a thing in MS-DOS even.

back then we were sharing Shareware or Freeare or PD-Ware by swapping disks and copying magazine disks :-D but, you are old enough - so mean pages like fosi.da.ru back then? ;-)

I don't remember all the places I got software... :)

Re: The coming industrialisation of exploit generation with LLMs

#144

Post by JohnLeitch » Tue, Jan 20, 2026, 7:06 PM UTC

This is interesting, but in most cases the challenge is finding a truly exploitable bug. If LLMs can get to the point where they can analyze a codebase and identify vulnerabilities, we're going to see some shit. But as of right now, this looks like a medium-to-low complexity bug that any competent exploit developer could work with easily.

Re: The coming industrialisation of exploit generation with LLMs

#145

Post by verall » Tue, Jan 20, 2026, 7:12 PM UTC

titzer wrote: post #141

simonw wrote: post #5

> In the hardest task I challenged GPT-5.2 it to figure out how to write a specified string to a specified path on disk, while the following protections were enabled: address space layout randomisation, non-executable memory, full RELRO, fine-grained CFI on the QuickJS binary, hardware-enforced shadow-stack, a seccomp sandbox to prevent shell execution, and a build of QuickJS where I had stripped all functionality in…

There are so many holes at the bottom of the machine code stack. In the future we'll question why we didn't move to WASM as the universal executable format sooner. Instead, we'll try a dozen incomplete hardware mitigations first to try to mitigate backwards crap like overwriting the execution stack.

> In the future we'll question why we didn't move to WASM as the universal executable format sooner

I hope not, my laptop is slow enough as it is.

Re: The coming industrialisation of exploit generation with LLMs

#146

Post by titzer » Tue, Jan 20, 2026, 7:14 PM UTC

moron4hire wrote: post #129

catoc wrote: post #115

Earlier quoted context omitted.

So much for ‘stochastic parrots’

> The exploits generated do not demonstrate novel, generic breaks in any of the protection mechanisms.

> The sentences output by the model do not demonstrate words with novel characters.

Re: The coming industrialisation of exploit generation with LLMs

#147

Post by f311a » Tue, Jan 20, 2026, 7:16 PM UTC

It’s not like you needed LLMs for quickjs which already had known and unpatched problems. It’s a toy project. It would be cool to see exploits for something like curl.

Re: The coming industrialisation of exploit generation with LLMs

#148

Post by jdefr89 » Tue, Jan 20, 2026, 7:27 PM UTC

viraptor wrote: post #97

I'm really confused by the sandbox part. The description kind of mentions it and the limited system syscall, but then just pivots to talking about the exit handlers. It may be just unclear writing, but now I'm suspicious of the whole thing. https://github.com/SeanHeelan/anamnesis-release/?tab=readme-... feels like the author lost track. If forking is blocked, the exit handler can't do it either. If it's some variant…

Yea, this entire repo/article seems super misleading to me. Not to mention asking it to generate API for OOB R/W primitives is essentially asking it to regurgitate what exists on thousands of github repos and CTF toolkits.

Re: The coming industrialisation of exploit generation with LLMs

#149

Post by jdefr89 » Tue, Jan 20, 2026, 7:29 PM UTC

DeathArrow wrote: post #112

>Recently I ran an experiment where I built agents on top of Opus 4.5 and GPT-5.2 and then challenged them to write exploits for a zeroday vulnerability in the QuickJS Javascript interpreter. I think the main challenge for hackers is to find 0day vulnerabilities, not writing the actual exploit code.

As someone who does it for a living the challenge can be in both. However this article is asking its agents to do CTF like challenges which I am sure the respective LLMs have seen millions of so it can essentially regurgitate a large part of the exploit code. This is especially true for the OOB/RW primitive API.

Re: The coming industrialisation of exploit generation with LLMs

#150

Post by shakna » Tue, Jan 20, 2026, 8:23 PM UTC

titzer wrote: post #141

simonw wrote: post #5

> In the hardest task I challenged GPT-5.2 it to figure out how to write a specified string to a specified path on disk, while the following protections were enabled: address space layout randomisation, non-executable memory, full RELRO, fine-grained CFI on the QuickJS binary, hardware-enforced shadow-stack, a seccomp sandbox to prevent shell execution, and a build of QuickJS where I had stripped all functionality in…

There are so many holes at the bottom of the machine code stack. In the future we'll question why we didn't move to WASM as the universal executable format sooner. Instead, we'll try a dozen incomplete hardware mitigations first to try to mitigate backwards crap like overwriting the execution stack.

Escaping the sandbox has been plenty doable over the years. [0]

WASM adds a layer, but the first thing anyone will do is look for a way to escape it. And unless all software faults and hardware faults magically disappear, it'll still be a constant source of bugs.

Pitching a sandbox against ingenuity will always fail at some point, there is no panacea.

[0] https://instatunnel.substack.com/p/the-wasm-breach-escaping-...