Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The coming industrialisation of exploit generation with LLMs

sean.heelan.io

Post reply on HN
Print view View original

81–90 of 174 posts

Re: The coming industrialisation of exploit generation with LLMs

#81

Post by underdeserver »

tptacek wrote: post #55
akoboldfrying wrote: post #54

Earlier quoted context omitted.

What I mean is: There will be bugs* in that pure Go implementation, and static linking means you're baking them in forever. Why is this preferable to dynamic linking? * It's likely that C implementations will have bugs related to dynamic memory allocation that are absent from the Go implementation, because Go is GCed while C is not. But it would be very surprising if there were no bugs at all in the Go implementation…

They're prioritizing memory corruption vulnerabilities, is the point of going to extremes to ensure there's no compiled C in their binaries.

You can have memory corruption in pure Go code, too.

Re: The coming industrialisation of exploit generation with LLMs

#82

Post by 0xDEAFBEAD »

0xDEAFBEAD wrote: post #75
rwmj wrote: post #12

Earlier quoted context omitted.

With the exploits, you can try them and they either work or they don't. An attacker is not especially interested in analysing why the successful ones work. With the CVE reports some poor maintainer has to go through and triage them, which is far more work, and very asymmetrical because the reporters can generate their spam reports in volume while each one requires detailed analysis.

It can't be too long before Claude Code is capable of replication + triage + suggested fixes...

BTW regarding "suggested fixes", an interesting attack would be to report a bug along with a prompt injection which will cause Claude to suggest inserting a vulnerability in the codebase in question. So, it's important to review bug-report-originated Claude suggestions extra carefully. (And watch for prompt injection attacks.)

Another thought is the reproducible builds become more valuable than ever, because it actually becomes feasible for lots and lots of devs to scan the entire codebase for vulns using an LLM and then verify reproducibility.

Re: The coming industrialisation of exploit generation with LLMs

#83

Post by tptacek »

underdeserver wrote: post #81
tptacek wrote: post #55

Earlier quoted context omitted.

They're prioritizing memory corruption vulnerabilities, is the point of going to extremes to ensure there's no compiled C in their binaries.

You can have memory corruption in pure Go code, too.

Uh huh. That's where all the Go memory corruption vulnerabilities come from!

Re: The coming industrialisation of exploit generation with LLMs

#84

Post by cookiengineer »

tptacek wrote: post #55
akoboldfrying wrote: post #54

Earlier quoted context omitted.

What I mean is: There will be bugs* in that pure Go implementation, and static linking means you're baking them in forever. Why is this preferable to dynamic linking? * It's likely that C implementations will have bugs related to dynamic memory allocation that are absent from the Go implementation, because Go is GCed while C is not. But it would be very surprising if there were no bugs at all in the Go implementation…

They're prioritizing memory corruption vulnerabilities, is the point of going to extremes to ensure there's no compiled C in their binaries.

It would be nice if there was something similar to the ebpf verifier, but for static C, so that loop mistakes, out of boundary mistakes and avoidable satisfiability problems are caught right in the compile step.

The reason I'm so avoidant to using C libraries at all cost is that the ecosystem doesn't prioritize maintenance or other forms of code quality in its distribution. If you have to go to great lengths of having e.g. header only libraries, then what's the point of using C99/C++ at all? Back when conan came out I had hopes for it, but meanwhile I gave up on the ecosystem.

Don't get me wrong, Rust is great for its use cases, too. I just chose the mutex hell as a personal preference over the wrapping hell.

Re: The coming industrialisation of exploit generation with LLMs

#85

Post by cookiengineer »

underdeserver wrote: post #81
tptacek wrote: post #55

Earlier quoted context omitted.

They're prioritizing memory corruption vulnerabilities, is the point of going to extremes to ensure there's no compiled C in their binaries.

You can have memory corruption in pure Go code, too.

Nobody claimed otherwise. You're interacting with a kernel that invented its own programming language based on macros, after all, instead of relying on a compiler for that.

What could go wrong with this, right?

/s

Re: The coming industrialisation of exploit generation with LLMs

#86

Post by 0xbadcafebee »

0xDEAFBEAD wrote: post #73
tptacek wrote: post #47

Earlier quoted context omitted.

"C executables" are most of the frontier of exploit development, which is why this is a meaningful model problem.

Can we fight fire with fire, and use LLMs to rewrite all the C in Rust?

Sure, but the LLMs will just chain 14 functions instead of 7. If all C code is rewritten in Rust tomorrow that still leaves all the other bug classes. Eliminating a bug class might have made human attacks harder, but now with LLMs the "hardness" factor is purely how much token money you have.

Re: The coming industrialisation of exploit generation with LLMs

#87

Post by 0xbadcafebee »

er4hn wrote: post #3

I think the author makes some interesting points, but I'm not that worried about this. These tools feel symmetric for defenders to use as well. There's an easy to see path that involves running "LLM Red Teams" in CI before merging code or major releases. The fact that it's a somewhat time expensive (I'm ignoring cost here on purpose) test makes it feel similar to fuzzing for where it would fit in a pipeline. New tool…

An LLM Red Team is going to be too expensive most people; an actual infosec company will need to write the prompts, vet them, etc. But you don't need that to find exploits if you're just a human sitting at a console trying things. The hackers still have the massive advantage of 1) time, 2) cost (it will cost them less than the defenders/Red-Team-As-a-SaaS), and 3) they only have to get lucky once.

Re: The coming industrialisation of exploit generation with LLMs

#88

Post by 0xbadcafebee »

tptacek wrote: post #27
simonw wrote: post #25

Earlier quoted context omitted.

I hadn't heard of Halvar Flake but evidently he's a well respected figure in security - https://ringzer0.training/advisory-board-thomas-dullien-halv... mentions "After working at Google Project Zero, he cofounded startup optimyze, which was acquired by Elastic Security in 2021" His co-founder on optimyze was Sean Heelan, the author of the OP.

Yes, Halvar Flake is pretty well respected in exploit dev circles.

Sure he can write exploits, but can he cool a beer really fast?

Re: The coming industrialisation of exploit generation with LLMs

#89

Post by energy123 »

pizlonator wrote: post #69
er4hn wrote: post #3

I think the author makes some interesting points, but I'm not that worried about this. These tools feel symmetric for defenders to use as well. There's an easy to see path that involves running "LLM Red Teams" in CI before merging code or major releases. The fact that it's a somewhat time expensive (I'm ignoring cost here on purpose) test makes it feel similar to fuzzing for where it would fit in a pipeline. New tool…

Not symmetric at all. There are countless bugs to fund. If the offender runs these tools, then any bug they find becomes a cyberweapon. If the defender runs these tools, they will not thwart the offender unless they find and fix all of the bugs. Any vs all is not symmetric

LLMs effectively move us from A to B:

A) 1 cyber security employee, 1 determined attacker

B) 100 cyber security employees, 100 determined attackers

Which is better for defender?

Re: The coming industrialisation of exploit generation with LLMs

#90

Post by eru »

cookiengineer wrote: post #44
rvz wrote: post #39

Earlier quoted context omitted.

Tells you all you need to know around how extremely weak a C executable like QuickJS is for LLMs to exploit. (If you as an infosec researcher prompt them correctly to find and exploit vulnerabilities). > Leak a libc Pointer via Use-After-Free. The exploit uses the vulnerability to leak a pointer to libc. I doubt Rust would save you here unless the binary has very limited calls to libc, but would be much harder for a…

The reason I value Go so much is because you have a fat dependency free binary that's just a bunch of syscalls when you use CGO_ENABLED=0. Combine that with a minimal docker container and you don't even need a shell or anything but the kernel in those images.

Yes, you can have docker container images that only contain the actual binary you want to run.

But if you are using a VM, you don't even need the Linux kernel: some systems let you compiler your program to run directly on the hypervisor.

See eg https://github.com/hermit-os/hermit-rs or https://mirage.io/

Post reply on HN