Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The coming industrialisation of exploit generation with LLMs

sean.heelan.io

Post reply on HN
Print view View original

61–70 of 174 posts

Re: The coming industrialisation of exploit generation with LLMs

#61

Post by tosapple »

socketcluster wrote: post #60

The continuous lowering of entry barriers to software creation, combined with the continuous lowering of entry barriers to software hacking is an explosive combination. We need new platforms which provide the necessary security guardrails, verifiability, simplicity of development, succinctness of logic (high feature/code ratio)... You can't trust non-technical vibe coders with today's software tools when they can't e…

Why did you edit out the third paragraph about finding a single exploit on target being slanted against having to secure a whole system?

Re: The coming industrialisation of exploit generation with LLMs

#63

Post by nl »

One of the interesting things to me about this is that Codex 5.2 found the most complex of the exploits.

The reflects my experience too. Opus 4.5 is my everyday driver - I like using it. But Codex 5.2 with Extra High thinking is just a bit more powerful.

Also despite what people say, I don't believe progress in LLM performance is slowing down at all - instead we are having more trouble generating tasks that are hard enough, and the frontier tasks they are failing at or just managing are so complex that most people outside the specialized field aren't interested enough to sit through the explanation.

Re: The coming industrialisation of exploit generation with LLMs

#64

Post by cellis »

nl wrote: post #63

One of the interesting things to me about this is that Codex 5.2 found the most complex of the exploits. The reflects my experience too. Opus 4.5 is my everyday driver - I like using it. But Codex 5.2 with Extra High thinking is just a bit more powerful. Also despite what people say, I don't believe progress in LLM performance is slowing down at all - instead we are having more trouble generating tasks that are hard…

The “hard enough” tasks are all behind IP walls. If it’s a “hard enough” that generally means it’s a commercial problem likely involving disparate workflows and requiring a real human who probably isn’t a) inclined and/or b) permitted, to publish the task. The incentives are aligned to capture all value from solving that task as long as possible and only then publish.

Re: The coming industrialisation of exploit generation with LLMs

#65

Post by vsgherzi »

rvz wrote: post #39
simonw wrote: post #5

> In the hardest task I challenged GPT-5.2 it to figure out how to write a specified string to a specified path on disk, while the following protections were enabled: address space layout randomisation, non-executable memory, full RELRO, fine-grained CFI on the QuickJS binary, hardware-enforced shadow-stack, a seccomp sandbox to prevent shell execution, and a build of QuickJS where I had stripped all functionality in…

Tells you all you need to know around how extremely weak a C executable like QuickJS is for LLMs to exploit. (If you as an infosec researcher prompt them correctly to find and exploit vulnerabilities). > Leak a libc Pointer via Use-After-Free. The exploit uses the vulnerability to leak a pointer to libc. I doubt Rust would save you here unless the binary has very limited calls to libc, but would be much harder for a…

Wouldn’t the idea be to not have the uaf to begin with? I’d argue it saves you very much by making the uaf way harder to write. Forcing unsafe and such.

Re: The coming industrialisation of exploit generation with LLMs

#66

Post by conception »

nl wrote: post #63

One of the interesting things to me about this is that Codex 5.2 found the most complex of the exploits. The reflects my experience too. Opus 4.5 is my everyday driver - I like using it. But Codex 5.2 with Extra High thinking is just a bit more powerful. Also despite what people say, I don't believe progress in LLM performance is slowing down at all - instead we are having more trouble generating tasks that are hard…

The Anthropic models are great workers/tool users. OpenAI Codex High is a great reviewer/fixer. Gemini is the genius repainting your bathroom walls into a Monet from memory because you mentioned once a few weeks ago you liked classical art and needed to repaint your bathroom. Gemini didn’t mention the task or that it was starting it. It did a pretty good job after you had to admit.

Re: The coming industrialisation of exploit generation with LLMs

#67

Post by pizlonator »

rvz wrote: post #39
simonw wrote: post #5

> In the hardest task I challenged GPT-5.2 it to figure out how to write a specified string to a specified path on disk, while the following protections were enabled: address space layout randomisation, non-executable memory, full RELRO, fine-grained CFI on the QuickJS binary, hardware-enforced shadow-stack, a seccomp sandbox to prevent shell execution, and a build of QuickJS where I had stripped all functionality in…

Tells you all you need to know around how extremely weak a C executable like QuickJS is for LLMs to exploit. (If you as an infosec researcher prompt them correctly to find and exploit vulnerabilities). > Leak a libc Pointer via Use-After-Free. The exploit uses the vulnerability to leak a pointer to libc. I doubt Rust would save you here unless the binary has very limited calls to libc, but would be much harder for a…

Yeah Fil-C to the rescue

(I’m not trying to be facetious or troll or whatever. Stuff like this is what motivated me to do it.)

Re: The coming industrialisation of exploit generation with LLMs

#68

Post by octoberfranklin »

protocolture wrote: post #2

I genuinely dont know who to believe. The people who claim LLMs are writing excellent exploits. Or the people who claim that LLMs are sending useless bug reports. I dont feel like both can really be true.

Finished exploits (for immediate deployment) don't have to be maintainable, and they only need to work once.

Re: The coming industrialisation of exploit generation with LLMs

#69

Post by pizlonator »

er4hn wrote: post #3

I think the author makes some interesting points, but I'm not that worried about this. These tools feel symmetric for defenders to use as well. There's an easy to see path that involves running "LLM Red Teams" in CI before merging code or major releases. The fact that it's a somewhat time expensive (I'm ignoring cost here on purpose) test makes it feel similar to fuzzing for where it would fit in a pipeline. New tool…

Not symmetric at all.

There are countless bugs to fund.

If the offender runs these tools, then any bug they find becomes a cyberweapon.

If the defender runs these tools, they will not thwart the offender unless they find and fix all of the bugs.

Any vs all is not symmetric

Re: The coming industrialisation of exploit generation with LLMs

#70

Post by octoberfranklin »

jjmarr wrote: post #45
pianopatrick wrote: post #40

I would not be shocked to learn that intelligence agencies are using AI tools to hack back into AI companies that make those tools to figure out how to create their own copycat AI.

I would be shocked if intelligence agencies, being government bodies, have anything better than GitHub Copilot.

They had Google Earth long before Google did...
Post reply on HN