Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The coming industrialisation of exploit generation with LLMs

sean.heelan.io

Post reply on HN
Print view View original

91–100 of 174 posts

Re: The coming industrialisation of exploit generation with LLMs

#91

Post by GoatInGrey »

protocolture wrote: post #2

I genuinely dont know who to believe. The people who claim LLMs are writing excellent exploits. Or the people who claim that LLMs are sending useless bug reports. I dont feel like both can really be true.

Both are true. Exploits are a very narrow problem with unambiguous success metrics. While also naturally complementing the ingrained persistence of LLMs. Bug reports are much more fuzzy by comparison with open-ended goals that lead to the LLMs metaphorically cheating on their homework to satisfy the prompter who doesn't know any better.

Re: The coming industrialisation of exploit generation with LLMs

#92

Post by adrianN »

0xbadcafebee wrote: post #86
0xDEAFBEAD wrote: post #73

Earlier quoted context omitted.

Can we fight fire with fire, and use LLMs to rewrite all the C in Rust?

Sure, but the LLMs will just chain 14 functions instead of 7. If all C code is rewritten in Rust tomorrow that still leaves all the other bug classes. Eliminating a bug class might have made human attacks harder, but now with LLMs the "hardness" factor is purely how much token money you have.

Llms are not magic. Fixing a large class of exploits makes exploitation harder.

Re: The coming industrialisation of exploit generation with LLMs

#93

Post by NitpickLawyer »

er4hn wrote: post #3

I think the author makes some interesting points, but I'm not that worried about this. These tools feel symmetric for defenders to use as well. There's an easy to see path that involves running "LLM Red Teams" in CI before merging code or major releases. The fact that it's a somewhat time expensive (I'm ignoring cost here on purpose) test makes it feel similar to fuzzing for where it would fit in a pipeline. New tool…

> These tools feel symmetric for defenders to use as well.

I don't think so. From a pure mathematical standpoint, you'd need better (or equal) results at avg@1 or maj@x, while the attacker needs just pass@x to succeed. That is, the red agent needs to work just once, while the blue agent needs to work all the time. Current agents are much better (20-30%) at pass@x than maj@x.

In real life that's why you sometimes see titles like "teenager hacks into multi-billion dollar company and installs crypto malware".

I do think that you're right in that we'll see improved security stance by using red v. blue agents "in a loop". But I also think that red has a mathematical advantage here.

Re: The coming industrialisation of exploit generation with LLMs

#94

Post by NitpickLawyer »

SchemaLoad wrote: post #14
rwmj wrote: post #12

Earlier quoted context omitted.

With the exploits, you can try them and they either work or they don't. An attacker is not especially interested in analysing why the successful ones work. With the CVE reports some poor maintainer has to go through and triage them, which is far more work, and very asymmetrical because the reporters can generate their spam reports in volume while each one requires detailed analysis.

There's been several notable posts where maintainers found there was no bug at all, or the example code did not even call code from their project and had just found running a python script can do things on your computer. Entirely AI generated Issue reports and examples wasting maintainer time.

That's because the user of the tool didn't go through the troubles to setup the env properly (as the author of the blog did). So what they got was a "story about a bug", but without verification.

The proper way to use these tools (like in other verifiable tasks such as math or coding) is to give them a feedback loop and an easily verifiable success criteria. In security exploitation you either capture the flag or not. It's very easy (and cheap) to verify. So you can leave these things to bang their tokens against a wall, and only look at their output once they capture the flag. Or they output something somewhere verifiable (e.g. echo "pwned" > /root/.flag)

Re: The coming industrialisation of exploit generation with LLMs

#96

Post by psychoslave »

digdugdirk wrote: post #38
er4hn wrote: post #3

I think the author makes some interesting points, but I'm not that worried about this. These tools feel symmetric for defenders to use as well. There's an easy to see path that involves running "LLM Red Teams" in CI before merging code or major releases. The fact that it's a somewhat time expensive (I'm ignoring cost here on purpose) test makes it feel similar to fuzzing for where it would fit in a pipeline. New tool…

That's not how complex systems work though? You say that these tools feel "symmetric" for defenders to use, but having both sides use the same tools immediately puts the defenders at a disadvantage in the "asymmetric warfare" context. The defensive side needs everything to go right, all the time. The offensive side only needs something to go wrong once.

That really depends of the offensive class. If that is a single group with some agenda, then that's just everyone spending much resources on creating solution no permanent actor in the game want actually to escalate into, just show they have the tools and skills.

It's probably more worrying as you get script kiddies on steroids which can spawn all around with same mindset as even the dumbest significant geopolitical actor out there.

Re: The coming industrialisation of exploit generation with LLMs

#97

Post by viraptor »

I'm really confused by the sandbox part. The description kind of mentions it and the limited system syscall, but then just pivots to talking about the exit handlers. It may be just unclear writing, but now I'm suspicious of the whole thing. https://github.com/SeanHeelan/anamnesis-release/?tab=readme-... feels like the author lost track.

If forking is blocked, the exit handler can't do it either. If it's some variant of execve, the sandbox is preserved so we didn't gain much.

Edit: ok, I get it! Missed the "Goal: write exactly "PWNED" to /tmp/pwned". Which makes the sandbox part way less interesting as implemented. It's just saying you can't shell out to do it, but there's no sandbox breakout at any point in the exploit.

Re: The coming industrialisation of exploit generation with LLMs

#98

Post by rwmj »

airza wrote: post #19
rwmj wrote: post #12

Earlier quoted context omitted.

With the exploits, you can try them and they either work or they don't. An attacker is not especially interested in analysing why the successful ones work. With the CVE reports some poor maintainer has to go through and triage them, which is far more work, and very asymmetrical because the reporters can generate their spam reports in volume while each one requires detailed analysis.

All the attackers I’ve known are extremely, pathologically interested in understanding why their exploits work.

I mean someone attacking systems at scale for profit.

Re: The coming industrialisation of exploit generation with LLMs

#99

Post by rightbyte »

NitpickLawyer wrote: post #93
er4hn wrote: post #3

I think the author makes some interesting points, but I'm not that worried about this. These tools feel symmetric for defenders to use as well. There's an easy to see path that involves running "LLM Red Teams" in CI before merging code or major releases. The fact that it's a somewhat time expensive (I'm ignoring cost here on purpose) test makes it feel similar to fuzzing for where it would fit in a pipeline. New tool…

> These tools feel symmetric for defenders to use as well. I don't think so. From a pure mathematical standpoint, you'd need better (or equal) results at avg@1 or maj@x, while the attacker needs just pass@x to succeed. That is, the red agent needs to work just once, while the blue agent needs to work all the time. Current agents are much better (20-30%) at pass@x than maj@x. In real life that's why you sometimes see…

>> These tools feel symmetric for defenders to use as well.

> I don't think so. From a pure mathematical standpoint, you'd need better (or equal) results at avg@1 or maj@x, while the attacker needs just pass@x to succeed.

Executing remote code is a choice not some sort of force of nature.

Timesharing systems are inherently not safe and way too much effort is put into claiming the stone from Sisyphus.

SaaS and complex centralized software need to go and that is way over due.

Re: The coming industrialisation of exploit generation with LLMs

#100

Post by supriyo-biswas »

cookiengineer wrote: post #84
tptacek wrote: post #55

Earlier quoted context omitted.

They're prioritizing memory corruption vulnerabilities, is the point of going to extremes to ensure there's no compiled C in their binaries.

It would be nice if there was something similar to the ebpf verifier, but for static C, so that loop mistakes, out of boundary mistakes and avoidable satisfiability problems are caught right in the compile step. The reason I'm so avoidant to using C libraries at all cost is that the ecosystem doesn't prioritize maintenance or other forms of code quality in its distribution. If you have to go to great lengths of havin…

I believe this is fil-c[1].

[1] https://fil-c.org/

Post reply on HN