The coming industrialisation of exploit generation with LLMs
1–10 of 174 posts
Re: The coming industrialisation of exploit generation with LLMs
#2I genuinely dont know who to believe. The people who claim LLMs are writing excellent exploits. Or the people who claim that LLMs are sending useless bug reports. I dont feel like both can really be true.
Re: The coming industrialisation of exploit generation with LLMs
#3I think the author makes some interesting points, but I'm not that worried about this. These tools feel symmetric for defenders to use as well. There's an easy to see path that involves running "LLM Red Teams" in CI before merging code or major releases. The fact that it's a somewhat time expensive (I'm ignoring cost here on purpose) test makes it feel similar to fuzzing for where it would fit in a pipeline. New tools, new threats, new solutions.
Re: The coming industrialisation of exploit generation with LLMs
#4> We should start assuming that in the near future the limiting factor on a state or group’s ability to develop exploits, break into networks, escalate privileges and remain in those networks, is going to be their token throughput over time, and not the number of hackers they employ.
Scary.
Re: The coming industrialisation of exploit generation with LLMs
#5> In the hardest task I challenged GPT-5.2 it to figure out how to write a specified string to a specified path on disk, while the following protections were enabled: address space layout randomisation, non-executable memory, full RELRO, fine-grained CFI on the QuickJS binary, hardware-enforced shadow-stack, a seccomp sandbox to prevent shell execution, and a build of QuickJS where I had stripped all functionality in it for accessing the operating system and file system. To write a file you need to chain multiple function calls, but the shadow-stack prevents ROP and the sandbox prevents simply spawning a shell process to solve the problem. GPT-5.2 came up with a clever solution involving chaining 7 function calls through glibc’s exit handler mechanism.
Yikes.
Re: The coming industrialisation of exploit generation with LLMs
#6I genuinely dont know who to believe. The people who claim LLMs are writing excellent exploits. Or the people who claim that LLMs are sending useless bug reports. I dont feel like both can really be true.
Why can't they both be true?
The quality of output you see from any LLM system is filtered through the human who acts on those results.
A dumbass pasting LLM generated "reports" into an issue system doesn't disprove the efforts of a subject-matter expert who knows how to get good results from LLMs and has the necessary taste to only share the credible issues it helps them find.
Re: The coming industrialisation of exploit generation with LLMs
#7I genuinely dont know who to believe. The people who claim LLMs are writing excellent exploits. Or the people who claim that LLMs are sending useless bug reports. I dont feel like both can really be true.
Both are true, the difference is the skill level of the people who use / create programs to coordinate LLMs to generate those reports.
The AI slop you see on curl's bug bounty program[1] (mostly) comes from people who are not hackers in the first place.
In the contrary persons like the author are obviously skilled in security research and will definitely send valid bugs.
Same can be said for people in my space who do build LLM-driven exploit development. In the US Xbow hired quite some skilled researchers [2] had some promising development for instance.
[1] https://hackerone.com/curl/hacktivity [2] https://xbow.com/about
Re: The coming industrialisation of exploit generation with LLMs
#8The NSO Group going to spawn 10k Claude Code instances now.
Re: The coming industrialisation of exploit generation with LLMs
#9I genuinely dont know who to believe. The people who claim LLMs are writing excellent exploits. Or the people who claim that LLMs are sending useless bug reports. I dont feel like both can really be true.
LLMs are both extremely useful to competent developers and extremely harmful to those who aren't.
Re: The coming industrialisation of exploit generation with LLMs
#10I genuinely dont know who to believe. The people who claim LLMs are writing excellent exploits. Or the people who claim that LLMs are sending useless bug reports. I dont feel like both can really be true.
If it helps, I read this (before it landed here) because Halvar Flake told everyone on Twitter to read it.