Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I used o3 to find a remote zeroday in the Linux SMB implementation

sean.heelan.io

Post reply on HN
Print view View original

41–50 of 232 posts

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#41

Post by kweingar »

nxobject wrote: post #33

A small thing, but I found the author's project-organization practices useful – creating individual .prompt files for system prompt, background information, and auxiliary instructions [1], and then running it through `llm`. It reveals how good LLM use, like any other engineering tool, requires good engineering thinking – methodical, and oriented around thoughtful specifications that balance design constraints – for b…

How do we benchmark these different methodologies?

It all seems like vibes-based incantations. "You are an expert at finding vulnerabilities." "Please report only real vulnerabilities, not any false positives." Organizing things with made-up HTML tags because the models seem to like that for some reason. Where does engineering come into it?

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#43

Post by p_ing »

Hilift wrote: post #8

Does the vulnerability exist in other implementations of SMB?

Implementations of SMB (Windows, Samba, macOS, ksmbd) are going to be different (macOS has a terrible implementation, even though AFP is being deprecated). At this level, it's doubtful that the code is shared among all implementations.

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#44

Post by nindalf »

kweingar wrote: post #41
nxobject wrote: post #33

A small thing, but I found the author's project-organization practices useful – creating individual .prompt files for system prompt, background information, and auxiliary instructions [1], and then running it through `llm`. It reveals how good LLM use, like any other engineering tool, requires good engineering thinking – methodical, and oriented around thoughtful specifications that balance design constraints – for b…

How do we benchmark these different methodologies? It all seems like vibes-based incantations. "You are an expert at finding vulnerabilities." "Please report only real vulnerabilities, not any false positives." Organizing things with made-up HTML tags because the models seem to like that for some reason. Where does engineering come into it?

The author is up front about the limitations of their prompt. They say

> In fact my entire system prompt is speculative in that I haven’t ran a sufficient number of evaluations to determine if it helps or hinders, so consider it equivalent to me saying a prayer, rather than anything resembling science or engineering. Once I have ran those evaluations I’ll let you know.

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#45

Post by Balooga »

b112 wrote: post #38
roncesvalles wrote: post #25

Earlier quoted context omitted.

A lot of money is all you need~

A lot of burned coal, is what. The "don't blame the victim" trope is valid in many contexts. This one application might be "hackers are attacking vital infrastructure, so we need to fund vulnerabilities first". And hackers use AI now, likely hacked into and for free, to discover vulnerabilities. So we must use AI! Therefore, the hackers are contributing to global warming. We, dear reader, are innocent.

Between $3k and $30k to solve a single ARC-AGI problem [1]. Not sure if "100 runs" makes this comparable.

[1] https://techcrunch.com/2025/04/02/openais-o3-model-might-be-...

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#46

Post by grg0 »

dehrmann wrote: post #34

Are there better tools for finding this? It feels like the sort of thing static analysis should reliably find, but it's in the Linux kernel, so you'd think either coding standards or tooling around these sorts of C bugs would be mature.

Not the expert in the area, but "classic static analysis" (for lack of a better term) and concurrency bugs doesn't really check. There are specific modeling tools for concurrency, and they are an entirely different beast than static analysis that requires notation and language support to describe what threads access what data when. Concurrency bugs in static analysis probably requires a level of context and understanding that an LLM can easily churn through.

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#47

Post by yellow_lead »

dehrmann wrote: post #34

Are there better tools for finding this? It feels like the sort of thing static analysis should reliably find, but it's in the Linux kernel, so you'd think either coding standards or tooling around these sorts of C bugs would be mature.

Some static analysis tools can detect use after free or memory leaks. But since this one requires reasoning about multiple threads, I think it would've been unlikely to be found by static analysis.

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#48

Post by sdoering »

b112 wrote: post #38
roncesvalles wrote: post #25

Earlier quoted context omitted.

A lot of money is all you need~

A lot of burned coal, is what. The "don't blame the victim" trope is valid in many contexts. This one application might be "hackers are attacking vital infrastructure, so we need to fund vulnerabilities first". And hackers use AI now, likely hacked into and for free, to discover vulnerabilities. So we must use AI! Therefore, the hackers are contributing to global warming. We, dear reader, are innocent.

So basically running a microwave for about 800 seconds, or a bit more than 13 minutes per model?

Oh my god - the world is gonna end. Too bad, we panicked because of exaggerated energy consumption numbers for using an LLM when doing individual work.

Yes - when a lot of people do a lot of prompting, these 0ne tenth of a second to 8 seconds of running the microwave per prompt adds up. But I strongly suggest, that we could all drop our energy consumption significantly using other means, instead of blaming the blog post's author about his energy consumption.

The "lot of burned coal" is probably not that much in this blog post's case given that 1 kWh is about 0.12 kg coal equivalent (and yes, I know that we need to burn more than that for 1kWh. Still not that much, compared to quite a few other human activities.

If you want to read up on it, James O'Donnell and Casey Crownhart try to pull together a detailed account of AI energy usage for MIT Technology Review.[1] I found that quite enlightening.

[1]: https://www.technologyreview.com/2025/05/20/1116327/ai-energ...

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#49

Post by ianbutler »

sebmellen wrote: post #28
ianbutler wrote: post #23

Earlier quoted context omitted.

We’ve been working on a system that increases signal to noise dramatically for finding bugs, we’ve at the same time been thoroughly benchmarking the entire popular software agents space for this We’ve found a wide range of results and we have a conference talk coming up soon where we’ll be releasing everything publicly so stay tuned for that itll be pretty illuminating on the state of the space Edit: confusing wordin…

Interesting. This is for Bismuth? I saw your pilot program link — what does that involve?

Yup! So we have multiple businesses working with us and for pilots its deploying the tool, providing feedback (we're connected over slack with all our partners for a direct line to us), and making sure the uses fit expectations for your business and working towards long term partnership.

We have several deployments in other peoples clouds right now as well as usage of our own cloud version, so we're flexible here.

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#50

Post by gizmodo59 »

KTibow wrote: post #31

> With o3 you get something that feels like a human-written bug report, condensed to just present the findings, whereas with Sonnet 3.7 you get something like a stream of thought, or a work log. This is likely because the author didn't give Claude a scratchpad or space to think, essentially forcing it to mix its thoughts with its report. I'd be interested to see if using the official thinking mechanism gives it enoug…

Having tried both I’d say o3 is in a league of it’s own compared to 3.7 or even Gemini 2.5 pro. The benchmarks may show not a lot of gain but that matters a lot when the task is very complex. What’s surprising is that they announced it last November and only now it’s released a month back now? (I’m guessing lots of safety took time but no idea). Can’t wait for o4!
Post reply on HN