I wonder if later challenges would be cheaper if summary of lesser challenges and solutions were also provided? Building up difficulty.
The coming industrialisation of exploit generation with LLMs
111–120 of 174 posts
Re: The coming industrialisation of exploit generation with LLMs
#112>Recently I ran an experiment where I built agents on top of Opus 4.5 and GPT-5.2 and then challenged them to write exploits for a zeroday vulnerability in the QuickJS Javascript interpreter.
I think the main challenge for hackers is to find 0day vulnerabilities, not writing the actual exploit code.
Re: The coming industrialisation of exploit generation with LLMs
#113I think the author makes some interesting points, but I'm not that worried about this. These tools feel symmetric for defenders to use as well. There's an easy to see path that involves running "LLM Red Teams" in CI before merging code or major releases. The fact that it's a somewhat time expensive (I'm ignoring cost here on purpose) test makes it feel similar to fuzzing for where it would fit in a pipeline. New tool…
For that matter is this in principle much different from a fuzzer?
Re: The coming industrialisation of exploit generation with LLMs
#114> In the hardest task I challenged GPT-5.2 it to figure out how to write a specified string to a specified path on disk, while the following protections were enabled: address space layout randomisation, non-executable memory, full RELRO, fine-grained CFI on the QuickJS binary, hardware-enforced shadow-stack, a seccomp sandbox to prevent shell execution, and a build of QuickJS where I had stripped all functionality in…
Tells you all you need to know around how extremely weak a C executable like QuickJS is for LLMs to exploit. (If you as an infosec researcher prompt them correctly to find and exploit vulnerabilities). > Leak a libc Pointer via Use-After-Free. The exploit uses the vulnerability to leak a pointer to libc. I doubt Rust would save you here unless the binary has very limited calls to libc, but would be much harder for a…
> Tells you all you need to know around how extremely weak a C executable like QuickJS is for LLMs to exploit. (If you as an infosec researcher prompt them correctly to find and exploit vulnerabilities).
Wouldn't GP's approach work with any other executable using libc? Python, Node, Rust, etc?
I fail to see what is specific to either C or QuickJS in the GP's approach.
Re: The coming industrialisation of exploit generation with LLMs
#115> In the hardest task I challenged GPT-5.2 it to figure out how to write a specified string to a specified path on disk, while the following protections were enabled: address space layout randomisation, non-executable memory, full RELRO, fine-grained CFI on the QuickJS binary, hardware-enforced shadow-stack, a seccomp sandbox to prevent shell execution, and a build of QuickJS where I had stripped all functionality in…
So much for ‘stochastic parrots’
Re: The coming industrialisation of exploit generation with LLMs
#116I genuinely dont know who to believe. The people who claim LLMs are writing excellent exploits. Or the people who claim that LLMs are sending useless bug reports. I dont feel like both can really be true.
These exploits were costing $50 of API credit each. If you receive 5001 issues from $100 in API spend on bug hunting and one of the issues cost $50 and the other 5000 cost one cent each, and they’re all visually indistinguishable using perfect grammar and familiar cyber security lingo; hard to find the dianond.
Once your exploit machine is good enough, you can start using stolen credentials to mine more exploits. This is going to be the new version of malware installing bitcoin miners.
Re: The coming industrialisation of exploit generation with LLMs
#117I think the author makes some interesting points, but I'm not that worried about this. These tools feel symmetric for defenders to use as well. There's an easy to see path that involves running "LLM Red Teams" in CI before merging code or major releases. The fact that it's a somewhat time expensive (I'm ignoring cost here on purpose) test makes it feel similar to fuzzing for where it would fit in a pipeline. New tool…
> These tools feel symmetric for defenders to use as well. I don't think so. From a pure mathematical standpoint, you'd need better (or equal) results at avg@1 or maj@x, while the attacker needs just pass@x to succeed. That is, the red agent needs to work just once, while the blue agent needs to work all the time. Current agents are much better (20-30%) at pass@x than maj@x. In real life that's why you sometimes see…
[deleted]
Re: The coming industrialisation of exploit generation with LLMs
#118Earlier quoted context omitted.
Awesome! What’s your strategy for migration of the entire world’s infrastructure to whatever you’re thinking about?
My strategy is to not use "the entire world's infrastructure" which makes it redundant. If enough people cancel their leftpad-as-a-Service subscription the server can be unplugged. (Yes I am somewhat hyperbolic and yes I see use for internet connected servers and clients. I argue against the SaaS driven centralization.)
I mean, yeah you can have the joy of being right from the heights of the hill you are standing upon. But It seems like you grasp the heart of problem being discussed.
How do we deal with the floods threatening those living in the valleys and slopes?
Re: The coming industrialisation of exploit generation with LLMs
#119> In the hardest task I challenged GPT-5.2 it to figure out how to write a specified string to a specified path on disk, while the following protections were enabled: address space layout randomisation, non-executable memory, full RELRO, fine-grained CFI on the QuickJS binary, hardware-enforced shadow-stack, a seccomp sandbox to prevent shell execution, and a build of QuickJS where I had stripped all functionality in…
Maybe we can remove mitigations. Every exploit you see is: First, find a vulnerability (the difficult part). Then, drill through five layers of ultimately ineffective "mitigations" (the tedious but almost always doable part).
Probabilistic mitigations work against probabilistic attacks, I guess - but exploit writers aren't random, they are directed, and they find the weaknesses.
Re: The coming industrialisation of exploit generation with LLMs
#120I genuinely dont know who to believe. The people who claim LLMs are writing excellent exploits. Or the people who claim that LLMs are sending useless bug reports. I dont feel like both can really be true.
Depends near entirely on the model being used. A bug report by Opus and a bug report from Gemma3 are not of the same caliber.