Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The coming industrialisation of exploit generation with LLMs

sean.heelan.io

Post reply on HN
Print view View original

21–30 of 174 posts

Re: The coming industrialisation of exploit generation with LLMs

#21

Post by simonw »

anonymous908213 wrote: post #11
simonw wrote: post #6

Earlier quoted context omitted.

Why can't they both be true? The quality of output you see from any LLM system is filtered through the human who acts on those results. A dumbass pasting LLM generated "reports" into an issue system doesn't disprove the efforts of a subject-matter expert who knows how to get good results from LLMs and has the necessary taste to only share the credible issues it helps them find.

They can't both be true if we're talking about the premise of the article, which is the subject of the headline and expounded upon prominently in the body: The Industrialisation of Intrusion By ‘industrialisation’ I mean that the ability of an organisation to complete a task will be limited by the number of tokens they can throw at that task. In order for a task to be ‘industrialised’ in this way it needs two things:…

My expectation is that any organization that attempts this will need subject matter experts to both setup and run the swarm of exploit finding agents for them.

Re: The coming industrialisation of exploit generation with LLMs

#22

Post by moyix »

protocolture wrote: post #17
simonw wrote: post #6

Earlier quoted context omitted.

Why can't they both be true? The quality of output you see from any LLM system is filtered through the human who acts on those results. A dumbass pasting LLM generated "reports" into an issue system doesn't disprove the efforts of a subject-matter expert who knows how to get good results from LLMs and has the necessary taste to only share the credible issues it helps them find.

Theres no filtering mentioned in the OP article. It claims GPT only created working useful exploits. If it can do that, it could also submit those exploits as perfectly as bug reports?

There is filtering mentioned, it's just not done by a human:

> I have written up the verification process I used for the experiments here, but the summary is: an exploit tends to involve building a capability to allow you to do something you shouldn’t be able to do. If, after running the exploit, you can do that thing, then you’ve won. For example, some of the experiments involved writing an exploit to spawn a shell from the Javascript process. To verify this the verification harness starts a listener on a particular local port, runs the Javascript interpreter and then pipes a command into it to run a command line utility that connects to that local port. As the Javascript interpreter has no ability to do any sort of network connections, or spawning of another process in normal execution, you know that if you receive the connect back then the exploit works as the shell that it started has run the command line utility you sent to it.

It is more work to build such "perfect" verifiers, and they don't apply to every vulnerability type (how do you write a Python script to detect a logic bug in an arbitrary application?), but for bugs like these where the exploit goal is very clear (exec code or write arbitrary content to a file) they work extremely well.

Re: The coming industrialisation of exploit generation with LLMs

#23

Post by simonw »

SchemaLoad wrote: post #14
rwmj wrote: post #12

Earlier quoted context omitted.

With the exploits, you can try them and they either work or they don't. An attacker is not especially interested in analysing why the successful ones work. With the CVE reports some poor maintainer has to go through and triage them, which is far more work, and very asymmetrical because the reporters can generate their spam reports in volume while each one requires detailed analysis.

There's been several notable posts where maintainers found there was no bug at all, or the example code did not even call code from their project and had just found running a python script can do things on your computer. Entirely AI generated Issue reports and examples wasting maintainer time.

My hunch is that the dumbasses submitting those reports were't actually using coding agent harnesses at all - they were pasting blocks of code into ChatGPT or other non-agent-harness tools and asking for vulnerabilities and reporting what came back.

An "agent harness" here is software that directly writes and executes code to test that it works. A vulnerability reported by such an agent harness with included proof-of-concept code that has been demonstrated to work is a different thing from an "exploit" that was reported by having a long context model spit out a bunch of random ideas based purely on reading the code.

I'm confident you can still find dumbasses who can mess up at using coding agent harnesses and create invalid, time wasting bug reports. Dumbasses are gonna dumbass.

Re: The coming industrialisation of exploit generation with LLMs

#24

Post by amelius »

er4hn wrote: post #3

I think the author makes some interesting points, but I'm not that worried about this. These tools feel symmetric for defenders to use as well. There's an easy to see path that involves running "LLM Red Teams" in CI before merging code or major releases. The fact that it's a somewhat time expensive (I'm ignoring cost here on purpose) test makes it feel similar to fuzzing for where it would fit in a pipeline. New tool…

> These tools feel symmetric for defenders to use as well.

Why? The attackers can run the defending software as well. As such they can test millions of testcases, and if one breaks through the defenses they can make it go live.

Re: The coming industrialisation of exploit generation with LLMs

#25

Post by simonw »

tptacek wrote: post #10
protocolture wrote: post #2

I genuinely dont know who to believe. The people who claim LLMs are writing excellent exploits. Or the people who claim that LLMs are sending useless bug reports. I dont feel like both can really be true.

If it helps, I read this (before it landed here) because Halvar Flake told everyone on Twitter to read it.

I hadn't heard of Halvar Flake but evidently he's a well respected figure in security - https://ringzer0.training/advisory-board-thomas-dullien-halv... mentions "After working at Google Project Zero, he cofounded startup optimyze, which was acquired by Elastic Security in 2021"

His co-founder on optimyze was Sean Heelan, the author of the OP.

Re: The coming industrialisation of exploit generation with LLMs

#27

Post by tptacek »

simonw wrote: post #25
tptacek wrote: post #10

Earlier quoted context omitted.

If it helps, I read this (before it landed here) because Halvar Flake told everyone on Twitter to read it.

I hadn't heard of Halvar Flake but evidently he's a well respected figure in security - https://ringzer0.training/advisory-board-thomas-dullien-halv... mentions "After working at Google Project Zero, he cofounded startup optimyze, which was acquired by Elastic Security in 2021" His co-founder on optimyze was Sean Heelan, the author of the OP.

Yes, Halvar Flake is pretty well respected in exploit dev circles.

Re: The coming industrialisation of exploit generation with LLMs

#28

Post by _carbyau_ »

My take away: apparently Cyberpunk Hackers of the dystopian future cruising through the virtual world will use GPT-5.2-or-greater as their "attack program" to break the "ICE" (Intrusion Countermeasures Electronics, not the currently politically charged term...).

I still doubt they will hook up their brains though.

Re: The coming industrialisation of exploit generation with LLMs

#29

Post by dfajgljsldkjag »

I was under the impression that once you have a vulnerability with code execution, writing the actual payload to exploit it is the easy part. With tools like pentools and etc is fairly straightforward.

The interesting part is still finding new potential RCE vulnerabilities, and generally if you can demonstrate the vulnerability even without demonstrating an E2E pwn red teams and white hats will still get credit.

Re: The coming industrialisation of exploit generation with LLMs

#30

Post by IanCal »

anonymous908213 wrote: post #11
simonw wrote: post #6

Earlier quoted context omitted.

Why can't they both be true? The quality of output you see from any LLM system is filtered through the human who acts on those results. A dumbass pasting LLM generated "reports" into an issue system doesn't disprove the efforts of a subject-matter expert who knows how to get good results from LLMs and has the necessary taste to only share the credible issues it helps them find.

They can't both be true if we're talking about the premise of the article, which is the subject of the headline and expounded upon prominently in the body: The Industrialisation of Intrusion By ‘industrialisation’ I mean that the ability of an organisation to complete a task will be limited by the number of tokens they can throw at that task. In order for a task to be ‘industrialised’ in this way it needs two things:…

A few points:

1. I think you have mixed up assistance and expertise. They talk about not needing a human in the loop for verification and to continue search but not about initial starts. Those are quite different. One well specified task can be attempted many times, and the skill sets are overlapping but not identical.

2. The article is about where they may get to rather than just what they are capable of now.

3. There’s no conflict between the idea that 10 parallel agents of the top models can mostly have one that successfully exploits a vulnerability - gated on an actual test that the exploit works - with feedback and iteration BUT random models pointed at arbitrary code without a good spec and without the ability to run code, and just run once, will generate lower quality results.

Post reply on HN