Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The coming industrialisation of exploit generation with LLMs

sean.heelan.io

Post reply on HN
Print view View original

161–170 of 174 posts

Re: The coming industrialisation of exploit generation with LLMs

#161

Post by 0xbadcafebee »

adrianN wrote: post #92
0xbadcafebee wrote: post #86

Earlier quoted context omitted.

Sure, but the LLMs will just chain 14 functions instead of 7. If all C code is rewritten in Rust tomorrow that still leaves all the other bug classes. Eliminating a bug class might have made human attacks harder, but now with LLMs the "hardness" factor is purely how much token money you have.

Llms are not magic. Fixing a large class of exploits makes exploitation harder.

They kind of are magic, that's the point. You can just tell them to look at every other bug class, and keep them churning on it until they find something. You can fast-forward through years of exploit research in a week. The "difficulty" of different bug classes is almost gone. (I think people underestimate just how many exploits are out there in other classes because they've been hyperfocused on the low-hanging fruit)

Re: The coming industrialisation of exploit generation with LLMs

#162

Post by saagarjha »

koakuma-chan wrote: post #153
saagarjha wrote: post #103

Earlier quoted context omitted.

Usually rewriting something in Rust requires nontrivial choices on the part of the translator that I’m not sure are currently within the reach of LLMs.

I heard this before, that apparently there are things you cannot implement in Rust. Like, apparently you cannot implement certain data structures in Rust. I think this is bullshit. Rust supports raw pointers, etc. You can implement whatever you want in Rust.

That’s not what I said. I am saying that translating C code to Rust usually involves a human in the loop because it requires non-trivial decisions to produce a good result.

Re: The coming industrialisation of exploit generation with LLMs

#163

Post by staticassertion »

simonw wrote: post #23
SchemaLoad wrote: post #14

Earlier quoted context omitted.

There's been several notable posts where maintainers found there was no bug at all, or the example code did not even call code from their project and had just found running a python script can do things on your computer. Entirely AI generated Issue reports and examples wasting maintainer time.

My hunch is that the dumbasses submitting those reports were't actually using coding agent harnesses at all - they were pasting blocks of code into ChatGPT or other non-agent-harness tools and asking for vulnerabilities and reporting what came back. An "agent harness" here is software that directly writes and executes code to test that it works. A vulnerability reported by such an agent harness with included proof-of…

I strongly suspect the same thing - that they weren't using agents at all in the reports we've seen, let alone agents with instructions on how to verify a viable attack, a threat model, etc.

Re: The coming industrialisation of exploit generation with LLMs

#164

Post by staticassertion »

underdeserver wrote: post #81
tptacek wrote: post #55

Earlier quoted context omitted.

They're prioritizing memory corruption vulnerabilities, is the point of going to extremes to ensure there's no compiled C in their binaries.

You can have memory corruption in pure Go code, too.

And in Rust (yes, safe Rust can have memory safety vulnerabilities). Who cares? They basically don't happen in practice.

Re: The coming industrialisation of exploit generation with LLMs

#165

Post by cookiengineer »

saagarjha wrote: post #102
cookiengineer wrote: post #84

Earlier quoted context omitted.

It would be nice if there was something similar to the ebpf verifier, but for static C, so that loop mistakes, out of boundary mistakes and avoidable satisfiability problems are caught right in the compile step. The reason I'm so avoidant to using C libraries at all cost is that the ecosystem doesn't prioritize maintenance or other forms of code quality in its distribution. If you have to go to great lengths of havin…

What do you consider to be a loop mistake?

Everything that is a "too clever" state management in an iterative loop.

Examples that come to mind: queues that are manipulated inside a loop, slice calls that forget to do length-- of the variable they set in the begin statement, char arrays that are overflowing because the loop doesn't check the length at the correct position in the code, conditions that are re-set inside the loop, like a min/max boundary that is set by an outer loop.

This kind of stuff. I guess you could argue these are memory safety issues. I've seen so crappy loop statements that the devs didn't bother to test it because they still believed they were "smart code", even after sending the devs a PoC that exploited their naive parser assumptions.

In Go I try to write clear, concise and "dumb" code so that a future me can still read it after years of not touching it. That's what I understand under Go's maintainability idiom, I suppose.

Re: The coming industrialisation of exploit generation with LLMs

#166

Post by saagarjha »

rramadass wrote: post #131
saagarjha wrote: post #109

> The exploits generated do not demonstrate novel, generic breaks in any of the protection mechanisms. They take advantage of known flaws in those protection mechanisms and gaps that exist in real deployments of them. These are the same gaps that human exploit developers take advantage of, as they also typically do not come up with novel breaks of exploit mitigations for each exploit. I actually think this result is…

> formal modeling of exploits, which is currently a very immature field. Can you elaborate more on this with pointers to some resources?

I think a lot of work that went into mitigating Spectre has been a good example since it’s very easy to patch incorrectly if you don’t have a good model of the vulnerability and what it allows

Re: The coming industrialisation of exploit generation with LLMs

#167

Post by ahartmetz »

staticassertion wrote: post #154
ahartmetz wrote: post #119

Earlier quoted context omitted.

Maybe we can remove mitigations. Every exploit you see is: First, find a vulnerability (the difficult part). Then, drill through five layers of ultimately ineffective "mitigations" (the tedious but almost always doable part). Probabilistic mitigations work against probabilistic attacks, I guess - but exploit writers aren't random, they are directed, and they find the weaknesses.

Most mitigations just flat out do not attempt to help against "arbitrary read/write". The LLM didn't just find "a vuln" and then work through the mitigations, it found the most powerful possible vulnerability. Lots of vulnerabilites get stopped dead by these mitigations. You almost always need multiple vulnerabilities tied together, which relies on a level of vulnerability density that's tractable. This is not just b…

Maybe I've been fooled by survivorship bias? You don't read much about the the vulnerabilities that ultimately weren't exploitable.

Reports about the ones that are exploitable usually read to me like after finding an entry, the attacker reaches into the well-stocked toolbox of post-entry techniques (return-oriented programming, nop slides, return to libc...) to do the rest of the work.

Re: The coming industrialisation of exploit generation with LLMs

#168

Post by socketcluster »

tosapple wrote: post #61
socketcluster wrote: post #60

The continuous lowering of entry barriers to software creation, combined with the continuous lowering of entry barriers to software hacking is an explosive combination. We need new platforms which provide the necessary security guardrails, verifiability, simplicity of development, succinctness of logic (high feature/code ratio)... You can't trust non-technical vibe coders with today's software tools when they can't e…

Why did you edit out the third paragraph about finding a single exploit on target being slanted against having to secure a whole system?

What I said was true but after thinking about it a bit more, I wasn't sure how material it was to my argument after considering additional factors.

There are other nuances which may offset the asymmetry a bit; for example the security analyst generally has much more visibility over the company's code than the hacker does.

That said, I stand by my original point because I think that building secure systems is really hard; it's much more effort per unit of functionality to build the system correctly (and doing that for every part of it) than it is to crack it (by finding a single hole).

On the side of defense, you need to understand a lot of nuance about how your system works and how parts interact to make it secure; any neglect can potentially be a critical vulnerability which can compromise the entire system.

On the side of offense, sometimes mindless prodding can uncover a critical vulnerability. The intelligence/thinking requirement is lower; it's more about knowledge than thinking.

For example, there are some special payloads which you can send which may pose a problem for different systems built by different companies because the companies share the same underlying engine or they fell victim to the same footgun. I think this aspect is much more important than my previous argument.

Re: The coming industrialisation of exploit generation with LLMs

#169

Post by 0xjnml »

Imustaskforhelp wrote: post #151
jerf wrote: post #130

Earlier quoted context omitted.

About a year ago I had some code I had been working on for about a year subject to a pretty heavy-duty security review by a reputable review company. When they asked what language I implemented it in and I told them "Go", they joked that half their job was done right there. While Go isn't perfect and you can certainly write some logic bugs that sufficiently clever use of a more strongly-typed language might let you a…

I love golang a lot and I feel like in this context of QuickJS it would be interesting to see what a port of QuickJS with Golang might look like security wise & a comparison to rust in the amount of security as well. Of course Golang and rust are apples to oranges comparison but still, if someone experienced in golang were to say port to QuickJS to golang and same for rust, aside from some performance cost which can…

See https://pkg.go.dev/modernc.org/quickjs

Re: The coming industrialisation of exploit generation with LLMs

#170

Post by jerf »

Imustaskforhelp wrote: post #151
jerf wrote: post #130

Earlier quoted context omitted.

About a year ago I had some code I had been working on for about a year subject to a pretty heavy-duty security review by a reputable review company. When they asked what language I implemented it in and I told them "Go", they joked that half their job was done right there. While Go isn't perfect and you can certainly write some logic bugs that sufficiently clever use of a more strongly-typed language might let you a…

I love golang a lot and I feel like in this context of QuickJS it would be interesting to see what a port of QuickJS with Golang might look like security wise & a comparison to rust in the amount of security as well. Of course Golang and rust are apples to oranges comparison but still, if someone experienced in golang were to say port to QuickJS to golang and same for rust, aside from some performance cost which can…

If you want to see what a JS interpreter in Go would look like, you can look at https://pkg.go.dev/github.com/robertkrimen/otto and https://github.com/dop251/goja . Of course they aren't "ports", but I feel like having two fairly complete intepreters is probably enough to prove the point. Arguably even a "port" would require enough changes that it wouldn't really be a "port" anyhow.

(The quickjs package in the sibling comment is the original compiled into C. It will probably have all the security quirks of the original as a result.)

Post reply on HN